($INBOX_DIR/description missing)help / color / mirror / Atom feed
[PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes 44+ messages / 3 participants [nested] [flat]
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v2 3/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 10 +- src/test/regress/sql/create_index.sql | 4 +- 7 files changed, 89 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cdc01c49c9..b82d122353 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3688,12 +3688,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 344cabaf52..66be1c9c8c 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2428,11 +2429,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2453,22 +2453,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2550,7 +2538,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2568,7 +2556,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2696,11 +2686,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2813,8 +2800,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3018,13 +3004,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3486,17 +3465,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 97cbaa3072..e72bbbb425 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c77c9a6ed5..eea6b01791 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..8ad103d80b 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,12 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..cd7f13b307 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,10 +903,10 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') -- 2.17.0 --JwB53PgKC5A7+0Ej-- ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v3 2/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cdc01c49c9..b82d122353 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3688,12 +3688,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 42c905867c..16cbf8a670 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2437,11 +2438,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2462,22 +2462,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2559,7 +2547,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2577,7 +2565,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2705,11 +2695,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2822,8 +2809,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3027,13 +3013,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3495,17 +3474,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 97cbaa3072..e72bbbb425 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --SCOJXUq1iwCn05li Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0003-Implement-CLUSTER-of-partitioned-table.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 7 +- src/backend/commands/indexcmds.c | 119 +++++++++++++-------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 ++-- src/test/regress/sql/create_index.sql | 13 ++- 7 files changed, 106 insertions(+), 68 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..7abd7f5cfc 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,7 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Avoid erroring out */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..df3e567acb 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,8 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); +static void ReindexPartitionedRel(Oid reloid, int options, bool concurrent, + bool isTopLevel); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2421,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2445,10 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) - { - ReindexPartitionedIndex(irel); - return; - } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) + ReindexPartitionedRel(indOid, options, concurrent, isTopLevel); + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2530,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2548,9 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + ReindexPartitionedRel(heapOid, options, concurrent, isTopLevel); + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2688,11 +2678,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2805,8 +2792,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +2996,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3457,71 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * ReindexPartitionedRel + * Reindex each child of the given partitioned relation. */ static void -ReindexPartitionedIndex(Relation parentIdx) +ReindexPartitionedRel(Oid reloid, int options, bool concurrent, bool isTopLevel) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + PreventInTransactionBlock(isTopLevel, + "REINDEX of partitioned relations"); // XXX + + /* + * Create list of children in longlived context, since we process each + * child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + MemoryContextSwitchTo(oldcontext); + + foreach (lc, inhoids) + { + Oid inhrelid = lfirst_oid(lc); + + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ + PushActiveSnapshot(GetTransactionSnapshot()); + + switch (get_rel_relkind(inhrelid)) + { + case RELKIND_PARTITIONED_INDEX: + case RELKIND_PARTITIONED_TABLE: + /* + * We have a full list of direct and indirect children, so skip + * partitioned relations and just handle their children. + */ + continue; + case RELKIND_INDEX: + reindex_index(inhrelid, false, get_rel_persistence(inhrelid), + options | REINDEXOPT_REPORT_PROGRESS); + break; + case RELKIND_RELATION: + (void) reindex_relation(inhrelid, + REINDEX_REL_PROCESS_TOAST | + REINDEX_REL_CHECK_CONSTRAINTS, + options | REINDEXOPT_REPORT_PROGRESS); + break; + } + } + + PopActiveSnapshot(); + CommitTransactionCommand(); + StartTransactionCommand(); } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --JYK4vJDZwFMowpUq Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes @ 2020-06-07 20:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Justin Pryzby @ 2020-06-07 20:59 UTC (permalink / raw) --- doc/src/sgml/ref/reindex.sgml | 5 - src/backend/catalog/index.c | 8 +- src/backend/commands/indexcmds.c | 137 ++++++++++++++------- src/backend/tcop/utility.c | 6 +- src/include/commands/defrem.h | 6 +- src/test/regress/expected/create_index.out | 18 +-- src/test/regress/sql/create_index.sql | 13 +- 7 files changed, 122 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index aac5d5be23..e2e32b3ba0 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -258,11 +258,6 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN can always reindex anything. </para> - <para> - Reindexing partitioned tables or partitioned indexes is not supported. - Each individual partition can be reindexed separately instead. - </para> - <refsect2 id="sql-reindex-concurrently" xreflabel="Rebuilding Indexes Concurrently"> <title>Rebuilding Indexes Concurrently</title> diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1be27eec52..f69f027890 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3660,12 +3660,8 @@ reindex_relation(Oid relid, int flags, int options) */ rel = table_open(relid, ShareLock); - /* - * This may be useful when implemented someday; but that day is not today. - * For now, avoid erroring out when called in a multi-table context - * (REINDEX SCHEMA) and happen to come across a partitioned table. The - * partitions may be reindexed on their own anyway. - */ + /* Skip partitioned tables if called in a multi-table context. The + * partitions are reindexed on their own. */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { ereport(WARNING, diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 2baca12c5f..68d75916ae 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -88,7 +88,10 @@ static List *ChooseIndexColumnNames(List *indexElems); static void RangeVarCallbackForReindexIndex(const RangeVar *relation, Oid relId, Oid oldRelId, void *arg); static bool ReindexRelationConcurrently(Oid relationOid, int options); -static void ReindexPartitionedIndex(Relation parentIdx); + +static List *PartitionedLeaves(Oid reloid); +static void ReindexMultipleRelsWorker(List *relids, int options, + bool concurrent); static void update_relispartition(Oid relationId, bool newval); static bool CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts); @@ -2420,11 +2423,10 @@ ChooseIndexColumnNames(List *indexElems) * Recreate a specific index. */ void -ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) +ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, bool isTopLevel) { struct ReindexIndexCallbackState state; Oid indOid; - Relation irel; char persistence; /* @@ -2445,22 +2447,13 @@ ReindexIndex(RangeVar *indexRelation, int options, bool concurrent) RangeVarCallbackForReindexIndex, &state); - /* - * Obtain the current persistence of the existing index. We already hold - * lock on the index. - */ - irel = index_open(indOid, NoLock); - - if (irel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) + persistence = get_rel_persistence(indOid); + if (get_rel_relkind(indOid) == RELKIND_PARTITIONED_INDEX) { - ReindexPartitionedIndex(irel); - return; + List *relids = PartitionedLeaves(indOid); + ReindexMultipleRelsWorker(relids, options, concurrent); } - - persistence = irel->rd_rel->relpersistence; - index_close(irel, NoLock); - - if (concurrent && persistence != RELPERSISTENCE_TEMP) + else if (concurrent && persistence != RELPERSISTENCE_TEMP) ReindexRelationConcurrently(indOid, options); else reindex_index(indOid, false, persistence, @@ -2542,7 +2535,7 @@ RangeVarCallbackForReindexIndex(const RangeVar *relation, * Recreate all indexes of a table (and of its toast table, if any) */ Oid -ReindexTable(RangeVar *relation, int options, bool concurrent) +ReindexTable(RangeVar *relation, int options, bool concurrent, bool isTopLevel) { Oid heapOid; bool result; @@ -2560,7 +2553,12 @@ ReindexTable(RangeVar *relation, int options, bool concurrent) 0, RangeVarCallbackOwnsTable, NULL); - if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) + if (get_rel_relkind(heapOid) == RELKIND_PARTITIONED_TABLE) + { + List *relids = PartitionedLeaves(heapOid); + ReindexMultipleRelsWorker(relids, options, concurrent); + } + else if (concurrent && get_rel_persistence(heapOid) != RELPERSISTENCE_TEMP) { result = ReindexRelationConcurrently(heapOid, options); @@ -2604,7 +2602,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, MemoryContext private_context; MemoryContext old; List *relids = NIL; - ListCell *l; int num_keys; bool concurrent_warning = false; @@ -2688,11 +2685,8 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * Only regular tables and matviews can have indexes, so ignore any * other kind of relation. * - * It is tempting to also consider partitioned tables here, but that - * has the problem that if the children are in the same schema, they - * would be processed twice. Maybe we could have a separate list of - * partitioned tables, and expand that afterwards into relids, - * ignoring any duplicates. + * Partitioned tables/indexes are skipped but matching leaf + * partitions are processed. */ if (classtuple->relkind != RELKIND_RELATION && classtuple->relkind != RELKIND_MATVIEW) @@ -2755,22 +2749,59 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, table_endscan(scan); table_close(relationRelation, AccessShareLock); - /* Now reindex each rel in a separate transaction */ + ReindexMultipleRelsWorker(relids, options, concurrent); + MemoryContextDelete(private_context); +} + +/* Reindex each rel in a separate transaction */ +static void +ReindexMultipleRelsWorker(List *relids, int options, bool concurrent) +{ + ListCell *l; + + /* + * This cannot run inside a user transaction block; if + * we were inside a transaction, then its commit- and + * start-transaction-command calls would not have the + * intended effect! + */ + // PreventInTransactionBlock(isTopLevel, + // "REINDEX of partitioned relations"); // XXX + PopActiveSnapshot(); CommitTransactionCommand(); + foreach(l, relids) { Oid relid = lfirst_oid(l); + char relkind; StartTransactionCommand(); + /* functions in indexes may want a snapshot set */ PushActiveSnapshot(GetTransactionSnapshot()); + relkind = get_rel_relkind(relid); + + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + { + abort(); + // ReindexPartitionedRel(relid, options, concurrent, true); + } + if (concurrent && get_rel_persistence(relid) != RELPERSISTENCE_TEMP) { + /* Handles concurrent indexing of both indexes and tables */ (void) ReindexRelationConcurrently(relid, options); /* ReindexRelationConcurrently() does the verbose output */ } + else if (relkind == RELKIND_INDEX) + { + reindex_index(relid, false, get_rel_persistence(relid), + options | REINDEXOPT_REPORT_PROGRESS); + PopActiveSnapshot(); + } else { bool result; @@ -2792,8 +2823,6 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, CommitTransactionCommand(); } StartTransactionCommand(); - - MemoryContextDelete(private_context); } @@ -2805,8 +2834,7 @@ ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, * view. For tables and materialized views, all its indexes will be rebuilt, * excluding invalid indexes and any indexes used in exclusion constraints, * but including its associated toast table indexes. For indexes, the index - * itself will be rebuilt. If 'relationOid' belongs to a partitioned table - * then we issue a warning to mention these are not yet supported. + * itself will be rebuilt. * * The locks taken on parent tables and involved indexes are kept until the * transaction is committed, at which point a session lock is taken on each @@ -3010,13 +3038,6 @@ ReindexRelationConcurrently(Oid relationOid, int options) MemoryContextSwitchTo(oldcontext); break; } - case RELKIND_PARTITIONED_TABLE: - /* see reindex_relation() */ - ereport(WARNING, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX of partitioned tables is not yet implemented, skipping \"%s\"", - get_rel_name(relationOid)))); - return false; default: /* Return error if type of relation is not supported */ ereport(ERROR, @@ -3478,17 +3499,41 @@ ReindexRelationConcurrently(Oid relationOid, int options) } /* - * ReindexPartitionedIndex - * Reindex each child of the given partitioned index. - * - * Not yet implemented. + * PartitionedLeaves + * Return a list of leaf partitions to be reindexed. */ -static void -ReindexPartitionedIndex(Relation parentIdx) +static List * +PartitionedLeaves(Oid reloid) { - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("REINDEX is not yet implemented for partitioned indexes"))); + MemoryContext oldcontext, reindex_context; + List *inhoids; + ListCell *lc; + + /* + * Create list of children in longlived context, since we will process + * each child in a separate transaction + */ + reindex_context = AllocSetContextCreate(PortalContext, "Reindex", + ALLOCSET_DEFAULT_SIZES); + oldcontext = MemoryContextSwitchTo(reindex_context); + inhoids = find_all_inheritors(reloid, NoLock, NULL); + + /* + * We have a full list of direct and indirect children, so remove from + * the list any partitioned relations (including the rel itself) and + * handle the leaves directly. + */ + foreach (lc, inhoids) + { + Oid relid = lfirst_oid(lc); + char relkind = get_rel_relkind(relid); + if (relkind == RELKIND_PARTITIONED_INDEX || + relkind == RELKIND_PARTITIONED_TABLE) + inhoids = foreach_delete_current(inhoids, lc); + } + + MemoryContextSwitchTo(oldcontext); + return inhoids; } /* diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 9b0c376c8c..fd6bc65c18 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -926,10 +926,12 @@ standard_ProcessUtility(PlannedStmt *pstmt, switch (stmt->kind) { case REINDEX_OBJECT_INDEX: - ReindexIndex(stmt->relation, stmt->options, stmt->concurrent); + ReindexIndex(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_TABLE: - ReindexTable(stmt->relation, stmt->options, stmt->concurrent); + ReindexTable(stmt->relation, stmt->options, + stmt->concurrent, isTopLevel); break; case REINDEX_OBJECT_SCHEMA: case REINDEX_OBJECT_SYSTEM: diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h index c26a102b17..df32f5b201 100644 --- a/src/include/commands/defrem.h +++ b/src/include/commands/defrem.h @@ -34,8 +34,10 @@ extern ObjectAddress DefineIndex(Oid relationId, bool check_not_in_use, bool skip_build, bool quiet); -extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent); -extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent); +extern void ReindexIndex(RangeVar *indexRelation, int options, bool concurrent, + bool isTopLevel); +extern Oid ReindexTable(RangeVar *relation, int options, bool concurrent, + bool isTopLevel); extern void ReindexMultipleTables(const char *objectName, ReindexObjectType objectKind, int options, bool concurrent); extern char *makeObjectName(const char *name1, const char *name2, diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index e3e6634d7e..61b4a30db4 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2196,18 +2196,20 @@ SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_ind concur_reindex_part_index_0_2 | concur_reindex_part_index_0 | 2 (5 rows) --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; -ERROR: REINDEX is not yet implemented for partitioned indexes --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; REINDEX TABLE concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes to reindex REINDEX TABLE CONCURRENTLY concur_reindex_part_10; -WARNING: REINDEX of partitioned tables is not yet implemented, skipping "concur_reindex_part_10" -NOTICE: table "concur_reindex_part_10" has no indexes that can be reindexed concurrently SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; relid | parentrelid | level diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql index f3667bacdc..a751dd5caa 100644 --- a/src/test/regress/sql/create_index.sql +++ b/src/test/regress/sql/create_index.sql @@ -903,12 +903,21 @@ CREATE INDEX concur_reindex_part_index_0_2 ON ONLY concur_reindex_part_0_2 (c1); ALTER INDEX concur_reindex_part_index_0 ATTACH PARTITION concur_reindex_part_index_0_2; SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; --- REINDEX fails for partitioned indexes +-- REINDEX for partitioned indexes +REINDEX INDEX concur_reindex_part_index; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index; +REINDEX INDEX concur_reindex_part_index_0; +REINDEX INDEX CONCURRENTLY concur_reindex_part_index_0; REINDEX INDEX concur_reindex_part_index_10; REINDEX INDEX CONCURRENTLY concur_reindex_part_index_10; --- REINDEX is a no-op for partitioned tables +-- REINDEX for partitioned tables REINDEX TABLE concur_reindex_part_10; REINDEX TABLE CONCURRENTLY concur_reindex_part_10; +REINDEX TABLE concur_reindex_part_0; +REINDEX TABLE CONCURRENTLY concur_reindex_part_0; +REINDEX TABLE concur_reindex_part_10; +REINDEX TABLE CONCURRENTLY concur_reindex_part_10; + SELECT relid, parentrelid, level FROM pg_partition_tree('concur_reindex_part_index') ORDER BY relid, level; -- REINDEX should preserve dependencies of partition tree. -- 2.17.0 --hYooF8G/hrfVAmum Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0002-Allow-CREATE-INDEX-CONCURRENTLY-on-partitioned-ta.patch" ^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: "box" type description @ 2023-11-01 15:36 Bruce Momjian <[email protected]> 2023-11-02 08:28 ` Re: "box" type description Kyotaro Horiguchi <[email protected]> 0 siblings, 1 reply; 44+ messages in thread From: Bruce Momjian @ 2023-11-01 15:36 UTC (permalink / raw) To: Christoph Berg <[email protected]>; Kyotaro Horiguchi <[email protected]>; pgsql-hackers On Wed, Mar 31, 2021 at 01:43:47PM +0200, Christoph Berg wrote: > Re: Kyotaro Horiguchi > > Returing to the description of pg_types, it should be changed like > > this following the discussion here. > > > > - pg_catalog | box | geometric box '(lower left,upper right)' > > + pg_catalog | box | geometric box 'lower left,upper right' > > > > But I find it hard to read. I fixed it instead as the following in the > > attached. However, it might rather be better not changing it.. > > > > + pg_catalog | box | geometric box 'pt-lower-left,pt-upper-right' > > I like that because it points to the "point" syntax so users can > figure out how to spell a box. I liked Horiguchi-san's patch from 2021, but once I started looking further, I found a number of improvements that can be made in the \dTS output beyond Horiguchi-san's changes: * boolean outputs 't'/'f', not 'true'/'false' * Added "format" ... for output * tid output format was at the start, not the end * I didn't add space between point x,y because the output has no space * I spelled out "point" instead of "pt" * "line" has two very different input formats so I listed both (more different than others like boolean) * I didn't see the need to say "datatype" for LSN and UUID * I improved the txid_snapshot description * There was no description for table_am_handler Patch attached. -- Bruce Momjian <[email protected]> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. Attachments: [text/x-diff] box.diff (7.9K, ../../[email protected]/2-box.diff) download | inline diff: diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 5a6cfbd94d..55340b00ad 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -3366,13 +3366,13 @@ SELECT person.name, holidays.num_weeks FROM person, holidays <entry><type>lseg</type></entry> <entry>32 bytes</entry> <entry>Finite line segment</entry> - <entry>((x1,y1),(x2,y2))</entry> + <entry>[(x1,y1),(x2,y2)]</entry> </row> <row> <entry><type>box</type></entry> <entry>32 bytes</entry> <entry>Rectangular box</entry> - <entry>((x1,y1),(x2,y2))</entry> + <entry>(x1,y1),(x2,y2)</entry> </row> <row> <entry><type>path</type></entry> diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 92bcaf2c73..6fd3c64e40 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -32,7 +32,7 @@ # OIDS 1 - 99 { oid => '16', array_type_oid => '1000', - descr => 'boolean, \'true\'/\'false\'', + descr => 'boolean, format \'t\'/\'f\'', typname => 'bool', typlen => '1', typbyval => 't', typcategory => 'B', typispreferred => 't', typinput => 'boolin', typoutput => 'boolout', typreceive => 'boolrecv', typsend => 'boolsend', typalign => 'c' }, @@ -90,7 +90,7 @@ typispreferred => 't', typinput => 'oidin', typoutput => 'oidout', typreceive => 'oidrecv', typsend => 'oidsend', typalign => 'i' }, { oid => '27', array_type_oid => '1010', - descr => '(block, offset), physical location of tuple', + descr => 'tuple physical location, format \'(block,offset)\'', typname => 'tid', typlen => '6', typbyval => 'f', typcategory => 'U', typinput => 'tidin', typoutput => 'tidout', typreceive => 'tidrecv', typsend => 'tidsend', typalign => 's' }, @@ -179,34 +179,34 @@ # OIDS 600 - 699 { oid => '600', array_type_oid => '1017', - descr => 'geometric point \'(x, y)\'', + descr => 'geometric point, format \'(x,y)\'', typname => 'point', typlen => '16', typbyval => 'f', typcategory => 'G', typsubscript => 'raw_array_subscript_handler', typelem => 'float8', typinput => 'point_in', typoutput => 'point_out', typreceive => 'point_recv', typsend => 'point_send', typalign => 'd' }, { oid => '601', array_type_oid => '1018', - descr => 'geometric line segment \'(pt1,pt2)\'', + descr => 'geometric line segment, format \'[point1,point2]\'', typname => 'lseg', typlen => '32', typbyval => 'f', typcategory => 'G', typsubscript => 'raw_array_subscript_handler', typelem => 'point', typinput => 'lseg_in', typoutput => 'lseg_out', typreceive => 'lseg_recv', typsend => 'lseg_send', typalign => 'd' }, { oid => '602', array_type_oid => '1019', - descr => 'geometric path \'(pt1,...)\'', + descr => 'geometric path, format \'(point1,...)\'', typname => 'path', typlen => '-1', typbyval => 'f', typcategory => 'G', typinput => 'path_in', typoutput => 'path_out', typreceive => 'path_recv', typsend => 'path_send', typalign => 'd', typstorage => 'x' }, { oid => '603', array_type_oid => '1020', - descr => 'geometric box \'(lower left,upper right)\'', + descr => 'geometric box, format \'lower left point,upper right point\'', typname => 'box', typlen => '32', typbyval => 'f', typcategory => 'G', typdelim => ';', typsubscript => 'raw_array_subscript_handler', typelem => 'point', typinput => 'box_in', typoutput => 'box_out', typreceive => 'box_recv', typsend => 'box_send', typalign => 'd' }, { oid => '604', array_type_oid => '1027', - descr => 'geometric polygon \'(pt1,...)\'', + descr => 'geometric polygon, format \'(point1,...)\'', typname => 'polygon', typlen => '-1', typbyval => 'f', typcategory => 'G', typinput => 'poly_in', typoutput => 'poly_out', typreceive => 'poly_recv', typsend => 'poly_send', typalign => 'd', typstorage => 'x' }, -{ oid => '628', array_type_oid => '629', descr => 'geometric line', +{ oid => '628', array_type_oid => '629', descr => 'geometric line, formats \'{A,B,C}\'/\'[point1,point2]\'', typname => 'line', typlen => '24', typbyval => 'f', typcategory => 'G', typsubscript => 'raw_array_subscript_handler', typelem => 'float8', typinput => 'line_in', typoutput => 'line_out', typreceive => 'line_recv', @@ -230,7 +230,7 @@ typcategory => 'X', typinput => 'unknownin', typoutput => 'unknownout', typreceive => 'unknownrecv', typsend => 'unknownsend', typalign => 'c' }, { oid => '718', array_type_oid => '719', - descr => 'geometric circle \'(center,radius)\'', + descr => 'geometric circle, format \'<center point,radius>\'', typname => 'circle', typlen => '24', typbyval => 'f', typcategory => 'G', typinput => 'circle_in', typoutput => 'circle_out', typreceive => 'circle_recv', typsend => 'circle_send', typalign => 'd' }, @@ -271,14 +271,14 @@ typinput => 'aclitemin', typoutput => 'aclitemout', typreceive => '-', typsend => '-', typalign => 'd' }, { oid => '1042', array_type_oid => '1014', - descr => 'char(length), blank-padded string, fixed storage length', + descr => '\'char(length)\' blank-padded string, fixed storage length', typname => 'bpchar', typlen => '-1', typbyval => 'f', typcategory => 'S', typinput => 'bpcharin', typoutput => 'bpcharout', typreceive => 'bpcharrecv', typsend => 'bpcharsend', typmodin => 'bpchartypmodin', typmodout => 'bpchartypmodout', typalign => 'i', typstorage => 'x', typcollation => 'default' }, { oid => '1043', array_type_oid => '1015', - descr => 'varchar(length), non-blank-padded string, variable storage length', + descr => '\'varchar(length)\' non-blank-padded string, variable storage length', typname => 'varchar', typlen => '-1', typbyval => 'f', typcategory => 'S', typinput => 'varcharin', typoutput => 'varcharout', typreceive => 'varcharrecv', typsend => 'varcharsend', @@ -345,7 +345,7 @@ # OIDS 1700 - 1799 { oid => '1700', array_type_oid => '1231', - descr => 'numeric(precision, decimal), arbitrary precision number', + descr => '\'numeric(precision, decimal)\' arbitrary precision number', typname => 'numeric', typlen => '-1', typbyval => 'f', typcategory => 'N', typinput => 'numeric_in', typoutput => 'numeric_out', typreceive => 'numeric_recv', typsend => 'numeric_send', @@ -400,13 +400,13 @@ typalign => 'i' }, # uuid -{ oid => '2950', array_type_oid => '2951', descr => 'UUID datatype', +{ oid => '2950', array_type_oid => '2951', descr => 'UUID', typname => 'uuid', typlen => '16', typbyval => 'f', typcategory => 'U', typinput => 'uuid_in', typoutput => 'uuid_out', typreceive => 'uuid_recv', typsend => 'uuid_send', typalign => 'c' }, # pg_lsn -{ oid => '3220', array_type_oid => '3221', descr => 'PostgreSQL LSN datatype', +{ oid => '3220', array_type_oid => '3221', descr => 'PostgreSQL LSN', typname => 'pg_lsn', typlen => '8', typbyval => 'FLOAT8PASSBYVAL', typcategory => 'U', typinput => 'pg_lsn_in', typoutput => 'pg_lsn_out', typreceive => 'pg_lsn_recv', typsend => 'pg_lsn_send', typalign => 'd' }, @@ -452,7 +452,7 @@ typreceive => 'jsonpath_recv', typsend => 'jsonpath_send', typalign => 'i', typstorage => 'x' }, -{ oid => '2970', array_type_oid => '2949', descr => 'txid snapshot', +{ oid => '2970', array_type_oid => '2949', descr => 'transaction snapshot', typname => 'txid_snapshot', typlen => '-1', typbyval => 'f', typcategory => 'U', typinput => 'txid_snapshot_in', typoutput => 'txid_snapshot_out', typreceive => 'txid_snapshot_recv', @@ -633,7 +633,9 @@ typoutput => 'tsm_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, { oid => '269', - typname => 'table_am_handler', typlen => '4', typbyval => 't', typtype => 'p', + typname => 'table_am_handler', + descr => 'pseudo-type for the result of a table AM handler function', + typlen => '4', typbyval => 't', typtype => 'p', typcategory => 'P', typinput => 'table_am_handler_in', typoutput => 'table_am_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, ^ permalink raw reply [nested|flat] 44+ messages in thread
* Re: "box" type description 2023-11-01 15:36 Re: "box" type description Bruce Momjian <[email protected]> @ 2023-11-02 08:28 ` Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 44+ messages in thread From: Kyotaro Horiguchi @ 2023-11-02 08:28 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; pgsql-hackers At Wed, 1 Nov 2023 11:36:01 -0400, Bruce Momjian <[email protected]> wrote in > On Wed, Mar 31, 2021 at 01:43:47PM +0200, Christoph Berg wrote: > > Re: Kyotaro Horiguchi > > I like that because it points to the "point" syntax so users can > > figure out how to spell a box. > > I liked Horiguchi-san's patch from 2021, but once I started looking > further, I found a number of improvements that can be made in the \dTS > output beyond Horiguchi-san's changes: > > * boolean outputs 't'/'f', not 'true'/'false' > * Added "format" ... for output > * tid output format was at the start, not the end > * I didn't add space between point x,y because the output has no space > * I spelled out "point" instead of "pt" > * "line" has two very different input formats so I listed both > (more different than others like boolean) > * I didn't see the need to say "datatype" for LSN and UUID > * I improved the txid_snapshot description > * There was no description for table_am_handler > > Patch attached. Thank you for continuing this. The additional changes looks fine. Upon reviewing the table again in this line, further potential improvements and issues have been found. For example: character, varchar: don't follow the rule. - 'char(length)' blank-padded string, fixed storage length + blank-padded string, fixed storage length, format 'char(length)' interval: doesn't follow the rule. - @ <number> <units>, time interval + time interval, format '[@] <number> <units>' (I think '@' is not necessary here..) pg_snapshot: The description given is just "snapshot", which seems overly simplistic. txid_snapshot: The description reads "transaction snapshot". Is this really accurate, especially in contrast with pg_snapshot? pg_brin_bloom_summary, pg_brin_minmax_multi_summary, pg_mcv_list and many: I'm uncertain whether these types, which lack an input syntax (but have an output format), qualify as pseudo-types. Nevertheless, I believe it would be beneficial to describe that those types differ from ordinary types. Should we consider refining these descriptions in the table? regards. -- Kyotaro Horiguchi NTT Open Source Software Center ^ permalink raw reply [nested|flat] 44+ messages in thread
end of thread, other threads:[~2023-11-02 08:28 UTC | newest] Thread overview: 44+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v2 3/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v3 2/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v4 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2020-06-07 20:59 [PATCH v5 1/3] Implement REINDEX of partitioned tables/indexes Justin Pryzby <[email protected]> 2023-11-01 15:36 Re: "box" type description Bruce Momjian <[email protected]> 2023-11-02 08:28 ` Re: "box" type description Kyotaro Horiguchi <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox