agora inbox for [email protected]
help / color / mirror / Atom feed[PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
80+ messages / 8 participants
[nested] [flat]
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 9492a3c6b9..0eda6678ac 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26448,12 +26448,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26538,6 +26539,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..aa66e6461f 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 93393fcfd4..911c12ee2c 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--mzh3msf6semlblkg
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Also add a new pg_index_has_outdated_dependency(regclass) SQL function, so
client code can filter such indexes if needed. This function will also be used
in a following commit to teach reindexdb to use this new OUTDATED option and
order the tables by the amount of work that will actually be done.
Catversion (should be) bumped.
Author: Julien Rouhaud <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/func.sgml | 27 ++++--
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/catalog/index.c | 107 ++++++++++++++++++++-
src/backend/commands/indexcmds.c | 12 ++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 40 ++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/catalog/index.h | 3 +
src/include/catalog/pg_proc.dat | 4 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
src/test/regress/expected/create_index.out | 10 ++
src/test/regress/sql/create_index.sql | 10 ++
13 files changed, 221 insertions(+), 12 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index bf99f82149..2cf6e66234 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -26381,12 +26381,13 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
<para>
<xref linkend="functions-admin-index-table"/> shows the functions
- available for index maintenance tasks. (Note that these maintenance
- tasks are normally done automatically by autovacuum; use of these
- functions is only required in special cases.)
- These functions cannot be executed during recovery.
- Use of these functions is restricted to superusers and the owner
- of the given index.
+ available for index maintenance tasks. (Note that the maintenance
+ tasks performing actions on indexes are normally done automatically by
+ autovacuum; use of these functions is only required in special cases.)
+ The functions performing actions on indexes cannot be executed during
+ recovery.
+ Use of the functions performing actions on indexes is restricted to
+ superusers and the owner of the given index.
</para>
<table id="functions-admin-index-table">
@@ -26471,6 +26472,20 @@ SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
option.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_index_has_outdated_dependency</primary>
+ </indexterm>
+ <function>pg_index_has_outdated_dependency</function> ( <parameter>index</parameter> <type>regclass</type> )
+ <returnvalue>boolean</returnvalue>
+ </para>
+ <para>
+ Check if the specified index has any outdated dependency. For now only
+ dependency on collations are supported.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index b22d39eba9..2d94d49cde 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ handle dependency is for the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..571feac5db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprected collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,105 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+Datum
+pg_index_has_outdated_dependency(PG_FUNCTION_ARGS)
+{
+ Oid indexOid = PG_GETARG_OID(0);
+ Relation rel;
+ bool isIndex;
+ bool res;
+
+ rel = try_relation_open(indexOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ isIndex = rel->rd_rel->relkind == RELKIND_INDEX;
+
+ if (!isIndex)
+ {
+ relation_close(rel, AccessShareLock);
+ PG_RETURN_NULL();
+ }
+
+ res = index_has_outdated_dependency(indexOid);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_BOOL(res);
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4096,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..0e1bcad101 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -3307,7 +3311,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3364,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..06545ee550 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,46 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ /*
+ * Check for outdated collation version dependency.
+ */
+ if (index_has_outdated_collation(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 9f0208ac49..fac3e0476a 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3d3974f467..4874d33996 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -949,6 +949,10 @@
proname => 'pg_indexam_has_property', provolatile => 's',
prorettype => 'bool', proargtypes => 'oid text',
prosrc => 'pg_indexam_has_property' },
+{ oid => '8102', descr => 'test property of an index',
+ proname => 'pg_index_has_outdated_dependency', provolatile => 's',
+ prorettype => 'bool', proargtypes => 'regclass',
+ prosrc => 'pg_index_has_outdated_dependency' },
{ oid => '637', descr => 'test property of an index',
proname => 'pg_index_has_property', provolatile => 's', prorettype => 'bool',
proargtypes => 'regclass text', prosrc => 'pg_index_has_property' },
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..5f7c49c650 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,16 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+NOTICE: table "reindex_coll" has no indexes to reindex
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..808cacee5d 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,16 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (OUTDATED) TABLE reindex_coll;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--oao5se3im4osqowp
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command.
@ 2020-12-03 07:54 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Julien Rouhaud @ 2020-12-03 07:54 UTC (permalink / raw)
OUTDATED is added a new unreserved keyword.
When used, REINDEX will only process indexes that have an outdated dependency.
For now, only dependency on collations are supported but we'll likely support
other kind of dependency in the future.
Author: Julien Rouhaud <[email protected]>
Reviewed-by: Michael Paquier, Mark Dilger
Discussion: https://postgr.es/m/20201203093143.GA64934%40nol
---
doc/src/sgml/ref/reindex.sgml | 12 +++
src/backend/access/index/indexam.c | 59 ++++++++++++++
src/backend/catalog/index.c | 79 ++++++++++++++++++-
src/backend/commands/indexcmds.c | 37 ++++++++-
src/backend/parser/gram.y | 4 +-
src/backend/utils/cache/relcache.c | 47 +++++++++++
src/bin/psql/tab-complete.c | 2 +-
src/include/access/genam.h | 1 +
src/include/catalog/index.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/utils/relcache.h | 1 +
.../regress/expected/collate.icu.utf8.out | 12 ++-
src/test/regress/expected/create_index.out | 27 +++++++
src/test/regress/sql/collate.icu.utf8.sql | 12 ++-
src/test/regress/sql/create_index.sql | 18 +++++
15 files changed, 301 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml
index ff4dba8c36..c4749c338b 100644
--- a/doc/src/sgml/ref/reindex.sgml
+++ b/doc/src/sgml/ref/reindex.sgml
@@ -26,6 +26,7 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
<phrase>where <replaceable class="parameter">option</replaceable> can be one of:</phrase>
CONCURRENTLY [ <replaceable class="parameter">boolean</replaceable> ]
+ OUTDATED [ <replaceable class="parameter">boolean</replaceable> ]
TABLESPACE <replaceable class="parameter">new_tablespace</replaceable>
VERBOSE [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
@@ -188,6 +189,17 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>OUTDATED</literal></term>
+ <listitem>
+ <para>
+ This option can be used to filter the list of indexes to rebuild and only
+ process indexes that have outdated dependencies. Fow now, the only
+ dependency handled currently is the collation provider version.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>TABLESPACE</literal></term>
<listitem>
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 3d2dbed708..dc1c85cf0d 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -145,6 +145,65 @@ index_open(Oid relationId, LOCKMODE lockmode)
return r;
}
+/* ----------------
+ * try_index_open - open an index relation by relation OID
+ *
+ * Same as index_open, except return NULL instead of failing
+ * if the index does not exist.
+ * ----------------
+ */
+Relation
+try_index_open(Oid relationId, LOCKMODE lockmode)
+{
+ Relation r;
+
+ Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES);
+
+ /* Get the lock first */
+ if (lockmode != NoLock)
+ LockRelationOid(relationId, lockmode);
+
+ /*
+ * Now that we have the lock, probe to see if the relation really exists
+ * or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relationId)))
+ {
+ /* Release useless lock */
+ if (lockmode != NoLock)
+ UnlockRelationOid(relationId, lockmode);
+
+ return NULL;
+ }
+
+ /* Should be safe to do a relcache load */
+ r = RelationIdGetRelation(relationId);
+
+ if (!RelationIsValid(r))
+ elog(ERROR, "could not open relation with OID %u", relationId);
+
+ /* If we didn't get the lock ourselves, assert that caller holds one */
+ Assert(lockmode != NoLock ||
+ CheckRelationLockedByMe(r, AccessShareLock, true));
+
+ if (r->rd_rel->relkind != RELKIND_INDEX &&
+ r->rd_rel->relkind != RELKIND_PARTITIONED_INDEX)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("\"%s\" is not an index",
+ RelationGetRelationName(r))));
+ }
+
+ /* Make note that we've accessed a temporary relation */
+ if (RelationUsesLocalBuffers(r))
+ MyXactFlags |= XACT_FLAGS_ACCESSEDTEMPNAMESPACE;
+
+ pgstat_initstats(r);
+
+ return r;
+}
+
/* ----------------
* index_close - close an index relation
*
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 4ef61b5efd..47e6a54149 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -100,6 +100,12 @@ typedef struct
Oid pendingReindexedIndexes[FLEXIBLE_ARRAY_MEMBER];
} SerializedReindexState;
+typedef struct
+{
+ Oid relid; /* targetr index oid */
+ bool outdated; /* depends on at least on deprecated collation? */
+} IndexHasOutdatedColl;
+
/* non-export function prototypes */
static bool relationHasPrimaryKey(Relation rel);
static TupleDesc ConstructTupleDescriptor(Relation heapRelation,
@@ -1351,6 +1357,77 @@ index_check_collation_versions(Oid relid)
list_free(context.warned_colls);
}
+/*
+ * Detect if an index depends on at least one outdated collation.
+ * This is a callback for visitDependenciesOf().
+ */
+static bool
+do_check_index_has_outdated_collation(const ObjectAddress *otherObject,
+ const char *version,
+ char **new_version,
+ void *data)
+{
+ IndexHasOutdatedColl *context = data;
+ char *current_version;
+
+ /* We only care about dependencies on collations. */
+ if (otherObject->classId != CollationRelationId)
+ return false;
+
+ /* Fast exit if we already found a outdated collation version. */
+ if (context->outdated)
+ return false;
+
+ /* Ask the provider for the current version. Give up if unsupported. */
+ current_version = get_collation_version_for_oid(otherObject->objectId,
+ false);
+ if (!current_version)
+ return false;
+
+ if (!version || strcmp(version, current_version) != 0)
+ context->outdated = true;
+
+ return false;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * collation version.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ */
+bool
+index_has_outdated_collation(Oid indexOid)
+{
+ ObjectAddress object;
+ IndexHasOutdatedColl context;
+
+ object.classId = RelationRelationId;
+ object.objectId = indexOid;
+ object.objectSubId = 0;
+
+ context.relid = indexOid;
+ context.outdated = false;
+
+ visitDependenciesOf(&object, &do_check_index_has_outdated_collation,
+ &context);
+
+ return context.outdated;
+}
+
+/*
+ * Check whether the given index has a dependency with an outdated
+ * refobjversion.
+ * Caller must hold a suitable lock and make sure that the given Oid belongs to
+ * an index.
+ * For now, only dependency on collations are supported.
+ */
+bool
+index_has_outdated_dependency(Oid indexOid)
+{
+ return index_has_outdated_collation(indexOid);
+}
+
/*
* Update the version for collations. A callback for visitDependenciesOf().
*/
@@ -3991,7 +4068,7 @@ reindex_relation(Oid relid, int flags, ReindexParams *params)
* relcache to get this with a sequential scan if ignoring system
* indexes.)
*/
- indexIds = RelationGetIndexList(rel);
+ indexIds = RelationGetIndexListFiltered(rel, params->options);
if (flags & REINDEX_REL_SUPPRESS_INDEX_USE)
{
diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c
index 8bc652ecd3..90ea95f40d 100644
--- a/src/backend/commands/indexcmds.c
+++ b/src/backend/commands/indexcmds.c
@@ -2484,6 +2484,7 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
bool concurrently = false;
bool verbose = false;
char *tablespacename = NULL;
+ bool outdated_filter = false;
/* Parse option list */
foreach(lc, stmt->params)
@@ -2496,6 +2497,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
concurrently = defGetBoolean(opt);
else if (strcmp(opt->defname, "tablespace") == 0)
tablespacename = defGetString(opt);
+ else if (strcmp(opt->defname, "outdated") == 0)
+ outdated_filter = defGetBoolean(opt);
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -2510,7 +2513,8 @@ ExecReindex(ParseState *pstate, ReindexStmt *stmt, bool isTopLevel)
params.options =
(verbose ? REINDEXOPT_VERBOSE : 0) |
- (concurrently ? REINDEXOPT_CONCURRENTLY : 0);
+ (concurrently ? REINDEXOPT_CONCURRENTLY : 0) |
+ (outdated_filter ? REINDEXOPT_OUTDATED : 0);
/*
* Assign the tablespace OID to move indexes to, with InvalidOid to do
@@ -2605,6 +2609,20 @@ ReindexIndex(RangeVar *indexRelation, ReindexParams *params, bool isTopLevel)
persistence = get_rel_persistence(indOid);
relkind = get_rel_relkind(indOid);
+ /*
+ * If the index isn't partitioned, we can detect here if it has any oudated
+ * dependency.
+ */
+ if (relkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(indOid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indOid))));
+ return;
+ }
+
if (relkind == RELKIND_PARTITIONED_INDEX)
ReindexPartitions(indOid, params, isTopLevel);
else if ((params->options & REINDEXOPT_CONCURRENTLY) != 0 &&
@@ -3049,6 +3067,17 @@ ReindexPartitions(Oid relid, ReindexParams *params, bool isTopLevel)
Assert(partkind == RELKIND_INDEX ||
partkind == RELKIND_RELATION);
+ /* Ignore indexes that don't depend on outdated dependency if needed */
+ if (partkind == RELKIND_INDEX &&
+ (params->options & REINDEXOPT_OUTDATED) != 0 &&
+ !index_has_outdated_dependency(partoid))
+ {
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(partoid))));
+ continue;
+ }
+
/* Save partition OID */
old_context = MemoryContextSwitchTo(reindex_context);
partitions = lappend_oid(partitions, partoid);
@@ -3307,7 +3336,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
RelationGetRelationName(heapRelation))));
/* Add all the valid indexes of relation to list */
- foreach(lc, RelationGetIndexList(heapRelation))
+ foreach(lc, RelationGetIndexListFiltered(heapRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc);
Relation indexRelation = index_open(cellOid,
@@ -3359,7 +3389,8 @@ ReindexRelationConcurrently(Oid relationOid, ReindexParams *params)
MemoryContextSwitchTo(oldcontext);
- foreach(lc2, RelationGetIndexList(toastRelation))
+ foreach(lc2, RelationGetIndexListFiltered(toastRelation,
+ params->options))
{
Oid cellOid = lfirst_oid(lc2);
Relation indexRelation = index_open(cellOid,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 652be0b96d..a67cfa867c 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
NULLS_P NUMERIC
OBJECT_P OF OFF OFFSET OIDS OLD ON ONLY OPERATOR OPTION OPTIONS OR
- ORDER ORDINALITY OTHERS OUT_P OUTER_P
+ ORDER ORDINALITY OTHERS OUT_P OUTDATED OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
PARALLEL PARSER PARTIAL PARTITION PASSING PASSWORD PLACING PLANS POLICY
@@ -15430,6 +15430,7 @@ unreserved_keyword:
| OPTIONS
| ORDINALITY
| OTHERS
+ | OUTDATED
| OVER
| OVERRIDING
| OWNED
@@ -16002,6 +16003,7 @@ bare_label_keyword:
| ORDINALITY
| OTHERS
| OUT_P
+ | OUTDATED
| OUTER_P
| OVERLAY
| OVERRIDING
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 7ef510cd01..94fc7b1c9e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -4620,6 +4620,53 @@ RelationGetIndexList(Relation relation)
return result;
}
+/*
+ * RelationGetIndexListFiltered -- get a filtered list of indexes on this
+ * relation.
+ *
+ * Calls RelationGetIndexList and only keep indexes that have an outdated
+ * dependency. For now, only collation version dependency is supported.
+ */
+List *
+RelationGetIndexListFiltered(Relation relation, bits32 options)
+{
+ List *result,
+ *full_list;
+ ListCell *lc;
+
+ full_list = RelationGetIndexList(relation);
+
+ /* Fast exit if no filtering was asked, or if the list if empty. */
+ if (((options & REINDEXOPT_OUTDATED) == 0) || full_list == NIL)
+ return full_list;
+
+ result = NIL;
+ foreach(lc, full_list)
+ {
+ Oid indexOid = lfirst_oid(lc);
+
+ Assert(get_rel_relkind(indexOid) == RELKIND_INDEX ||
+ get_rel_relkind(indexOid) == RELKIND_PARTITIONED_INDEX);
+
+ /*
+ * Check for any oudated dependency.
+ */
+ if (index_has_outdated_dependency(indexOid))
+ {
+ result = lappend_oid(result, indexOid);
+ continue;
+ }
+
+ /* Didn't find any outdated dependency, index will be ignored. */
+ if (((options & REINDEXOPT_OUTDATED) != 0))
+ ereport(NOTICE,
+ (errmsg("index \"%s\" has no outdated dependency",
+ get_rel_name(indexOid))));
+ }
+
+ return result;
+}
+
/*
* RelationGetStatExtList
* get a list of OIDs of statistics objects on this relation
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index ecdb8d752b..0843f36580 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3674,7 +3674,7 @@ psql_completion(const char *text, int start, int end)
* one word, so the above test is correct.
*/
if (ends_with(prev_wd, '(') || ends_with(prev_wd, ','))
- COMPLETE_WITH("CONCURRENTLY", "TABLESPACE", "VERBOSE");
+ COMPLETE_WITH("CONCURRENTLY", "OUTDATED'", "TABLESPACE", "VERBOSE");
else if (TailMatches("TABLESPACE"))
COMPLETE_WITH_QUERY(Query_for_list_of_tablespaces);
}
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 4515401869..fab5dbc101 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -138,6 +138,7 @@ typedef struct IndexOrderByDistance
#define IndexScanIsValid(scan) PointerIsValid(scan)
extern Relation index_open(Oid relationId, LOCKMODE lockmode);
+extern Relation try_index_open(Oid relationId, LOCKMODE lockmode);
extern void index_close(Relation relation, LOCKMODE lockmode);
extern bool index_insert(Relation indexRelation,
diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h
index e22d506436..298c0c633c 100644
--- a/src/include/catalog/index.h
+++ b/src/include/catalog/index.h
@@ -42,6 +42,7 @@ typedef struct ReindexParams
#define REINDEXOPT_REPORT_PROGRESS 0x02 /* report pgstat progress */
#define REINDEXOPT_MISSING_OK 0x04 /* skip missing relations */
#define REINDEXOPT_CONCURRENTLY 0x08 /* concurrent mode */
+#define REINDEXOPT_OUTDATED 0x10/* outdated collation only */
/* state info for validate_index bulkdelete callback */
typedef struct ValidateIndexState
@@ -137,6 +138,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo,
bool *isnull);
extern void index_check_collation_versions(Oid relid);
+extern bool index_has_outdated_collation(Oid indexOid);
+extern bool index_has_outdated_dependency(Oid indexOid);
extern void index_update_collation_versions(Oid relid, Oid coll);
extern void index_build(Relation heapRelation,
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 28083aaac9..e6c725d9a6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -295,6 +295,7 @@ PG_KEYWORD("order", ORDER, RESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("ordinality", ORDINALITY, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("others", OTHERS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("out", OUT_P, COL_NAME_KEYWORD, BARE_LABEL)
+PG_KEYWORD("outdated", OUTDATED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("over", OVER, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD, AS_LABEL)
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 2fcdf79323..a7a2272abd 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -45,6 +45,7 @@ extern void RelationClose(Relation relation);
*/
extern List *RelationGetFKeyList(Relation relation);
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetIndexListFiltered(Relation relation, bits32 options);
extern List *RelationGetStatExtList(Relation relation);
extern Oid RelationGetPrimaryKeyIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out
index de70cb1212..001f091fb3 100644
--- a/src/test/regress/expected/collate.icu.utf8.out
+++ b/src/test/regress/expected/collate.icu.utf8.out
@@ -2094,9 +2094,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
@@ -2107,9 +2109,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
objid
-------
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 830fdddf24..959fad6b71 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -2018,6 +2018,33 @@ INFO: index "reindex_verbose_pkey" was reindexed
\set VERBOSITY default
DROP TABLE reindex_verbose;
--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes to reindex
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+NOTICE: table "reindex_coll_1" has no indexes that can be reindexed concurrently
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+NOTICE: index "reindex_coll_1_pkey" has no outdated dependency
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+--
-- REINDEX CONCURRENTLY
--
CREATE TABLE concur_reindex_tab (c1 int);
diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql
index dd5d208854..a6504512f7 100644
--- a/src/test/regress/sql/collate.icu.utf8.sql
+++ b/src/test/regress/sql/collate.icu.utf8.sql
@@ -836,9 +836,11 @@ WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE collate_test;
-REINDEX TABLE collate_part_0;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE collate_test;
+REINDEX (OUTDATED) TABLE collate_part_0;
REINDEX TABLE collate_part_1;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
@@ -847,9 +849,11 @@ UPDATE pg_depend SET refobjversion = 'not a version'
WHERE refclassid = 'pg_collation'::regclass
AND objid::regclass::text LIKE 'icuidx%'
AND refobjversion IS NOT NULL;
-REINDEX TABLE CONCURRENTLY collate_test;
+SET client_min_messages to WARNING; -- pg_toast index names aren't stable
+REINDEX (OUTDATED) TABLE CONCURRENTLY collate_test;
REINDEX TABLE CONCURRENTLY collate_part_0;
-REINDEX INDEX CONCURRENTLY icuidx17_part;
+REINDEX (OUTDATED) INDEX CONCURRENTLY icuidx17_part;
+RESET client_min_messages;
SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version';
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index 8bc76f7c6f..4da6e2b5e8 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -790,6 +790,24 @@ REINDEX (VERBOSE) TABLE reindex_verbose;
\set VERBOSITY default
DROP TABLE reindex_verbose;
+--
+-- REINDEX (OUTDATED)
+--
+CREATE TABLE reindex_coll(id integer primary key) PARTITION BY LIST (id);
+CREATE TABLE reindex_coll_1 PARTITION OF reindex_coll FOR VALUES IN (1);
+\set VERBOSITY terse \\ -- suppress machine-dependent details
+-- no suitable index should be found
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED) INDEX reindex_coll_1_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_pkey;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) TABLE reindex_coll_1;
+REINDEX (VERBOSE, OUTDATED, CONCURRENTLY) INDEX reindex_coll_1_pkey;
+\set VERBOSITY default
+DROP TABLE reindex_coll;
+
--
-- REINDEX CONCURRENTLY
--
--
2.30.1
--rpt4sswdwkhtp4fw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-Add-a-outdated-option-to-reindexdb.patch"
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: optimize hashjoin
@ 2024-08-22 12:08 bucoo <[email protected]>
2024-08-22 12:14 ` Re: optimize hashjoin Kirill Reshke <[email protected]>
2024-08-22 19:26 ` Re: optimize hashjoin Tomas Vondra <[email protected]>
0 siblings, 2 replies; 80+ messages in thread
From: bucoo @ 2024-08-22 12:08 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; pgsql-hackers; +Cc: 'Tom Lane' <[email protected]>; 'Robert Haas' <[email protected]>
------=_Part_35779_1338760973.1724328493004
Content-Type: text/html;charset=utf-8
Content-Transfer-Encoding: base64
PHA+PGJyLz4mZ3Q7IDApIFRoZSBwYXRjaCBkb2VzIG5vdCBhcHBseSBhbnltb3JlLCB0aGFua3Mg
dG8gRGF2aWQgY29tbWl0dGluZyBhIHBhdGNoPC9wPjxwPiZndDsgeWVzdGVyZGF5LiBBdHRhY2hl
ZCBpcyBhIHBhdGNoIHJlYmFzZWQgb24gdG9wIG9mIGN1cnJlbnQgbWFzdGVyLjwvcD48cD5UaGF0
IHBhdGNoIGlzIGJhc2VkIG9uIFBHMTcuIEkgaGF2ZSBub3cgcmV3cml0dGVuIGl0IGJhc2VkIG9u
IHRoZSBtYXN0ZXIgYnJhbmNoIGFuZCBhZGRlZCBzb21lIGNvbW1lbnRzLjwvcD48cD48YnIvPiZn
dDsgMSkgV291bGRuJiMzOTt0IGl0IGJlIGVhc2llciAoYW5kIGp1c3QgYXMgZWZmaWNpZW50KSB0
byB1c2Ugc2xvdHMgd2l0aDwvcD48cD4mZ3Q7IFRUU09wc01pbmltYWxUdXBsZSwgaS5lLiBiYWNr
ZWQgYnkgYSBtaW5pbWFsIHR1cGxlPzwvcD48cD5Vc2UgZGlmZmVudCBraW5kIG9mIHNsb3QsIHRo
ZSBFeGVjRXZhbEV4cHIgZnVuY3Rpb24gd2lsbCByZXBvcnQgYW4gZXJyb3IuPGJyLz48L3A+PHA+
PGJyLz4KICZndDsgMikgSSBmaW5kIHRoZSBuYW1pbmcgb2YgdGhlIG5ldyBmdW5jdGlvbnMgYSBi
aXQgY29uZnVzaW5nLiBXZSBub3cgaGF2ZTxici8+Jmd0OyB0aGUgJnF1b3Q7b3JpZ2luYWwmcXVv
dDsgZnVuY3Rpb25zIHdvcmtpbmcgd2l0aCBzbG90cywgYW5kIHRoZW4gYWxzbyBmdW5jdGlvbnM8
YnIvPiZndDsgd2l0aCAmcXVvdDtUdXBsZSZxdW90OyB3b3JraW5nIHdpdGggdHVwbGVzLiBTZWVt
cyBhc3ltbWV0cmljLjwvcD48cD5JbiBuZXQgcGF0Y2ggZnVuY3Rpb24gbmFtZSByZW5hbWVkIHRv
IEV4ZWNIYXNoVGFibGVJbnNlcnRTbG90IGFuZCBFeGVjSGFzaFRhYmxlSW5zZXJ0VHVwbGUsPC9w
PjxwPmFsc28gRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0U2xvdEN1cnJlbnRCYXRjaCBhbmQg
RXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0VHVwbGVDdXJyZW50QmF0Y2guPGJyLz48L3A+PHA+
PGJyLz4KICZndDsgMykgVGhlIGNvZGUgaW4gRXhlY0hhc2hKb2luT3V0ZXJHZXRUdXBsZSBpcyBo
YXJkIHRvIHVuZGVyc3RhbmQsIGl0JiMzOTtkPGJyLz4mZ3Q7IHZlcnkgbXVjaCBiZW5lZml0IGZy
b20gc29tZSBjb21tZW50cy4gSSYjMzk7bSBhIGJpdCB1bnN1cmUgaWYgdGhlIGxpa2VseSgpPGJy
Lz4mZ3Q7IGFuZCB1bmxpa2VseSgpIGhpbnRzIHJlYWxseSBoZWxwLjxici8+PC9wPjxwPkluIG5l
dyBwYXRjaCBhZGRlZCBzb21lIGNvbW1lbnRzLjwvcD48cD4mcXVvdDtMaWtlbHkmcXVvdDsgYW5k
ICZxdW90O3VubGlrZWx5JnF1b3Q7IG1pZ2h0IG9mZmVyIG9ubHkgbWFyZ2luYWwgaGVscCBvbiBz
b21lIENQVXMgYW5kIG1pZ2h0IG5vdCBiZSBiZW5lZmljaWFsIGF0IGFsbCBvbiBvdGhlciBwbGF0
Zm9ybXMgKEkgdGhpbmspLjwvcD48cD48YnIvPgogJmd0OyA0KSBJcyB0aGUgaGpfb3V0ZXJUdXBs
ZUJ1ZmZlciBidWZmZXIgcmVhbGx5IG5lZWRlZCAvIGVmZmVjdGl2ZT8gSSYjMzk7ZCBiZXQ8YnIv
PiZndDsganVzdCB1c2luZyBwYWxsb2MoKSB3aWxsIHdvcmsganVzdCBhcyB3ZWxsLCB0aGFua3Mg
dG8gdGhlIEFsbG9jU2V0PGJyLz4mZ3Q7IGNhY2hpbmcgZXRjLjxici8+PC9wPjxwPlRoZSBoal9v
dXRlclR1cGxlQnVmZmVyIGF2b2lkIHJlZm9ybShtYXRlcmlhbGl6ZSkgdHVwbGUgaW4gbm9uLVRU
U09wc01pbmltYWxUdXBsZSBzY2VuYXJpb3MsPC9wPjxwPnNlZSBFeGVjRm9yY2VTdG9yZU1pbmlt
YWxUdXBsZS4gSSBhZGRlZCBzb21lIGNvbW1lbnRzIGluIG5ldyBwYXRjaC48YnIvPjwvcD48cD48
YnIvPiZndDsgQ2FuIHlvdSBwcm92aWRlIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgdGhlIGJlbmNo
bWFyayB5b3UgZGlkPyBXaGF0PGJyLz4mZ3Q7IGhhcmR3YXJlLCB3aGF0IHNjYWxlLCBQb3N0Z3Jl
U1FMIGNvbmZpZ3VyYXRpb24sIHdoaWNoIG9mIHRoZSAyMiBxdWVyaWVzPGJyLz4mZ3Q7IGFyZSBp
bXByb3ZlZCwgZXRjLjxici8+Jmd0OyA8YnIvPiZndDsgSSByYW4gVFBDLUggd2l0aCAxR0IgYW5k
IDEwR0Igc2NhbGVzIG9uIHR3byBtYWNoaW5lcywgYW5kIEkgc2VlIHByZXR0eTxici8+Jmd0OyBt
dWNoIG5vIGRpZmZlcmVuY2UgY29tcGFyZWQgdG8gbWFzdGVyLiBIb3dldmVyLCBpdCBvY2N1cnJl
ZCB0byBtZSB0aGU8YnIvPiZndDsgcGF0Y2ggb25seSBldmVyIGhlbHBzIGlmIHdlIGluY3JlYXNl
IHRoZSBudW1iZXIgb2YgYmF0Y2hlcyBkdXJpbmc8YnIvPiZndDsgZXhlY3V0aW9uLCBpbiB3aGlj
aCBjYXNlIHdlIG5lZWQgdG8gbW92ZSB0dXBsZXMgdG8gdGhlIHJpZ2h0IGJhdGNoLjwvcD48cD5P
bmx5IHBhcmFsbGVsIEhhc2hKb2luIHNwZWVkIHVwIHRvIH4yeChhbGwgZGF0YSBjYWNoZWQgaW4g
bWVtb3J5KSw8L3A+PHA+bm90IGZ1bGwgcXVlcnksIGluY2x1ZGUgbm9uLXBhcmFsbGVsIEhhc2hK
b2luLjwvcD48cD5ub24tcGFyYWxsZWwgSGFzaEpvaW4gb25seSB3aGVuIGJhdGNocyBsYXJnZSB0
aGVuIG9uZSB3aWxsIHNwZWVkIHVwLDwvcD48cD5iZWNhdXNlIHRoaXMgcGF0Y2ggb25seSBvcHRp
bWl6ZSBmb3IgcmVhZCBiYXRjaHMgdHVwbGVzIHRvIG1lbW9yeS48YnIvPjwvcD48YnIvPjxici8+
PGJyLz48ZGl2PjxhIGhyZWY9Imh0dHBzOi8vYS5hcHAucXEuY29tL28vc2ltcGxlLmpzcD9wa2du
YW1lPWNvbS5zb2h1Lm1haWwuY2xpZW50LmNvcmRvdmEiPjxpbWcgIGJvcmRlcj0iMCIgc3JjPSIv
LzFjYmJiMmQxNDg3NTMuY2RuLnNvaHVjcy5jb20vbWFpbC9pbWFnZXMvc2NvcmVfYWRfZm9vdDFf
NzUweDc5LnBuZyIvPiA8L2E+PC9kaXY+Cjxici8+PGJyLz48YnIvPg==
------=_Part_35779_1338760973.1724328493004
Content-Type: application/octet-stream; name=optimize-hashjoin-master.patch
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=optimize-hashjoin-master.patch
Content-Length: 12172
ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2V4ZWN1dG9yL25vZGVIYXNoLmMgYi9zcmMvYmFja2Vu
ZC9leGVjdXRvci9ub2RlSGFzaC5jCmluZGV4IDU3MGE5MGViZTEuLjVlNzAyYTM3ZmYgMTAwNjQ0
Ci0tLSBhL3NyYy9iYWNrZW5kL2V4ZWN1dG9yL25vZGVIYXNoLmMKKysrIGIvc3JjL2JhY2tlbmQv
ZXhlY3V0b3Ivbm9kZUhhc2guYwpAQCAtMTkwLDcgKzE5MCw3IEBAIE11bHRpRXhlY1ByaXZhdGVI
YXNoKEhhc2hTdGF0ZSAqbm9kZSkKIAkJCWVsc2UKIAkJCXsKIAkJCQkvKiBOb3Qgc3ViamVjdCB0
byBza2V3IG9wdGltaXphdGlvbiwgc28gaW5zZXJ0IG5vcm1hbGx5ICovCi0JCQkJRXhlY0hhc2hU
YWJsZUluc2VydChoYXNodGFibGUsIHNsb3QsIGhhc2h2YWx1ZSk7CisJCQkJRXhlY0hhc2hUYWJs
ZUluc2VydFNsb3QoaGFzaHRhYmxlLCBzbG90LCBoYXNodmFsdWUpOwogCQkJfQogCQkJaGFzaHRh
YmxlLT50b3RhbFR1cGxlcyArPSAxOwogCQl9CkBAIC0xNTk0LDcgKzE1OTQsNyBAQCBFeGVjUGFy
YWxsZWxIYXNoSW5jcmVhc2VOdW1CdWNrZXRzKEhhc2hKb2luVGFibGUgaGFzaHRhYmxlKQogfQog
CiAvKgotICogRXhlY0hhc2hUYWJsZUluc2VydAorICogRXhlY0hhc2hUYWJsZUluc2VydFNsb3QK
ICAqCQlpbnNlcnQgYSB0dXBsZSBpbnRvIHRoZSBoYXNoIHRhYmxlIGRlcGVuZGluZyBvbiB0aGUg
aGFzaCB2YWx1ZQogICoJCWl0IG1heSBqdXN0IGdvIHRvIGEgdGVtcCBmaWxlIGZvciBsYXRlciBi
YXRjaGVzCiAgKgpAQCAtMTYwNSwxMiArMTYwNSwyOSBAQCBFeGVjUGFyYWxsZWxIYXNoSW5jcmVh
c2VOdW1CdWNrZXRzKEhhc2hKb2luVGFibGUgaGFzaHRhYmxlKQogICogd29ydGggdGhlIG1lc3Np
bmVzcyByZXF1aXJlZC4KICAqLwogdm9pZAotRXhlY0hhc2hUYWJsZUluc2VydChIYXNoSm9pblRh
YmxlIGhhc2h0YWJsZSwKLQkJCQkJVHVwbGVUYWJsZVNsb3QgKnNsb3QsCi0JCQkJCXVpbnQzMiBo
YXNodmFsdWUpCitFeGVjSGFzaFRhYmxlSW5zZXJ0U2xvdChIYXNoSm9pblRhYmxlIGhhc2h0YWJs
ZSwKKwkJCQkJCVR1cGxlVGFibGVTbG90ICpzbG90LAorCQkJCQkJdWludDMyIGhhc2h2YWx1ZSkK
IHsKIAlib29sCQlzaG91bGRGcmVlOwogCU1pbmltYWxUdXBsZSB0dXBsZSA9IEV4ZWNGZXRjaFNs
b3RNaW5pbWFsVHVwbGUoc2xvdCwgJnNob3VsZEZyZWUpOworCisJRXhlY0hhc2hUYWJsZUluc2Vy
dFR1cGxlKGhhc2h0YWJsZSwgdHVwbGUsIGhhc2h2YWx1ZSk7CisKKwlpZiAoc2hvdWxkRnJlZSkK
KwkJaGVhcF9mcmVlX21pbmltYWxfdHVwbGUodHVwbGUpOworfQorCisvKgorICogRXhlY0hhc2hU
YWJsZUluc2VydFR1cGxlCisgKgkJaW5zZXJ0IGEgdHVwbGUgaW50byB0aGUgaGFzaCB0YWJsZSBk
ZXBlbmRpbmcgb24gdGhlIGhhc2ggdmFsdWUKKyAqCQlpdCBtYXkganVzdCBnbyB0byBhIHRlbXAg
ZmlsZSBmb3IgbGF0ZXIgYmF0Y2hlcworICovCit2b2lkCitFeGVjSGFzaFRhYmxlSW5zZXJ0VHVw
bGUoSGFzaEpvaW5UYWJsZSBoYXNodGFibGUsCisJCQkJCQkgTWluaW1hbFR1cGxlIHR1cGxlLAor
CQkJCQkJIHVpbnQzMiBoYXNodmFsdWUpCit7CiAJaW50CQkJYnVja2V0bm87CiAJaW50CQkJYmF0
Y2hubzsKIApAQCAtMTY4NSw5ICsxNzAyLDYgQEAgRXhlY0hhc2hUYWJsZUluc2VydChIYXNoSm9p
blRhYmxlIGhhc2h0YWJsZSwKIAkJCQkJCQkgICZoYXNodGFibGUtPmlubmVyQmF0Y2hGaWxlW2Jh
dGNobm9dLAogCQkJCQkJCSAgaGFzaHRhYmxlKTsKIAl9Ci0KLQlpZiAoc2hvdWxkRnJlZSkKLQkJ
aGVhcF9mcmVlX21pbmltYWxfdHVwbGUodHVwbGUpOwogfQogCiAvKgpAQCAtMTc2MSwxMiArMTc3
NSwxMCBAQCByZXRyeToKICAqIHR1cGxlcyB0aGF0IGJlbG9uZyBpbiB0aGUgY3VycmVudCBiYXRj
aCBvbmNlIGdyb3d0aCBoYXMgYmVlbiBkaXNhYmxlZC4KICAqLwogdm9pZAotRXhlY1BhcmFsbGVs
SGFzaFRhYmxlSW5zZXJ0Q3VycmVudEJhdGNoKEhhc2hKb2luVGFibGUgaGFzaHRhYmxlLAotCQkJ
CQkJCQkJCVR1cGxlVGFibGVTbG90ICpzbG90LAotCQkJCQkJCQkJCXVpbnQzMiBoYXNodmFsdWUp
CitFeGVjUGFyYWxsZWxIYXNoVGFibGVJbnNlcnRUdXBsZUN1cnJlbnRCYXRjaChIYXNoSm9pblRh
YmxlIGhhc2h0YWJsZSwKKwkJCQkJCQkJCQkJIE1pbmltYWxUdXBsZSB0dXBsZSwKKwkJCQkJCQkJ
CQkJIHVpbnQzMiBoYXNodmFsdWUpCiB7Ci0JYm9vbAkJc2hvdWxkRnJlZTsKLQlNaW5pbWFsVHVw
bGUgdHVwbGUgPSBFeGVjRmV0Y2hTbG90TWluaW1hbFR1cGxlKHNsb3QsICZzaG91bGRGcmVlKTsK
IAlIYXNoSm9pblR1cGxlIGhhc2hUdXBsZTsKIAlkc2FfcG9pbnRlciBzaGFyZWQ7CiAJaW50CQkJ
YmF0Y2hubzsKQEAgLTE3ODIsNiArMTc5NCwyMSBAQCBFeGVjUGFyYWxsZWxIYXNoVGFibGVJbnNl
cnRDdXJyZW50QmF0Y2goSGFzaEpvaW5UYWJsZSBoYXNodGFibGUsCiAJSGVhcFR1cGxlSGVhZGVy
Q2xlYXJNYXRjaChISlRVUExFX01JTlRVUExFKGhhc2hUdXBsZSkpOwogCUV4ZWNQYXJhbGxlbEhh
c2hQdXNoVHVwbGUoJmhhc2h0YWJsZS0+YnVja2V0cy5zaGFyZWRbYnVja2V0bm9dLAogCQkJCQkJ
CSAgaGFzaFR1cGxlLCBzaGFyZWQpOworfQorCisvKgorICogbGlrZSBFeGVjUGFyYWxsZWxIYXNo
VGFibGVJbnNlcnRUdXBsZUN1cnJlbnRCYXRjaCwKKyAqIGJ1dCB0aGlzIGZ1bmN0aW9uIGFjY2Vw
dCBhIFR1cGxlVGFibGVTbG90CisgKi8KK3ZvaWQKK0V4ZWNQYXJhbGxlbEhhc2hUYWJsZUluc2Vy
dFNsb3RDdXJyZW50QmF0Y2goSGFzaEpvaW5UYWJsZSBoYXNodGFibGUsCisJCQkJCQkJCQkJVHVw
bGVUYWJsZVNsb3QgKnNsb3QsCisJCQkJCQkJCQkJdWludDMyIGhhc2h2YWx1ZSkKK3sKKwlib29s
CQlzaG91bGRGcmVlOworCU1pbmltYWxUdXBsZSB0dXBsZSA9IEV4ZWNGZXRjaFNsb3RNaW5pbWFs
VHVwbGUoc2xvdCwgJnNob3VsZEZyZWUpOworCisJRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0
VHVwbGVDdXJyZW50QmF0Y2goaGFzaHRhYmxlLCB0dXBsZSwgaGFzaHZhbHVlKTsKIAogCWlmIChz
aG91bGRGcmVlKQogCQloZWFwX2ZyZWVfbWluaW1hbF90dXBsZSh0dXBsZSk7CkBAIC0yNDU0LDcg
KzI0ODEsNyBAQCBFeGVjSGFzaEdldFNrZXdCdWNrZXQoSGFzaEpvaW5UYWJsZSBoYXNodGFibGUs
IHVpbnQzMiBoYXNodmFsdWUpCiAgKgkJSW5zZXJ0IGEgdHVwbGUgaW50byB0aGUgc2tldyBoYXNo
dGFibGUuCiAgKgogICogVGhpcyBzaG91bGQgZ2VuZXJhbGx5IG1hdGNoIHVwIHdpdGggdGhlIGN1
cnJlbnQtYmF0Y2ggY2FzZSBpbgotICogRXhlY0hhc2hUYWJsZUluc2VydC4KKyAqIEV4ZWNIYXNo
VGFibGVJbnNlcnRTbG90LgogICovCiBzdGF0aWMgdm9pZAogRXhlY0hhc2hTa2V3VGFibGVJbnNl
cnQoSGFzaEpvaW5UYWJsZSBoYXNodGFibGUsCkBAIC0yNTM0LDggKzI1NjEsOCBAQCBFeGVjSGFz
aFJlbW92ZU5leHRTa2V3QnVja2V0KEhhc2hKb2luVGFibGUgaGFzaHRhYmxlKQogCQlTaXplCQl0
dXBsZVNpemU7CiAKIAkJLyoKLQkJICogVGhpcyBjb2RlIG11c3QgYWdyZWUgd2l0aCBFeGVjSGFz
aFRhYmxlSW5zZXJ0LiAgV2UgZG8gbm90IHVzZQotCQkgKiBFeGVjSGFzaFRhYmxlSW5zZXJ0IGRp
cmVjdGx5IGFzIEV4ZWNIYXNoVGFibGVJbnNlcnQgZXhwZWN0cyBhCisJCSAqIFRoaXMgY29kZSBt
dXN0IGFncmVlIHdpdGggRXhlY0hhc2hUYWJsZUluc2VydFNsb3QuICBXZSBkbyBub3QgdXNlCisJ
CSAqIEV4ZWNIYXNoVGFibGVJbnNlcnRTbG90IGRpcmVjdGx5IGFzIEV4ZWNIYXNoVGFibGVJbnNl
cnRTbG90IGV4cGVjdHMgYQogCQkgKiBUdXBsZVRhYmxlU2xvdCB3aGlsZSB3ZSBhbHJlYWR5IGhh
dmUgSGFzaEpvaW5UdXBsZXMuCiAJCSAqLwogCQl0dXBsZSA9IEhKVFVQTEVfTUlOVFVQTEUoaGFz
aFR1cGxlKTsKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2V4ZWN1dG9yL25vZGVIYXNoam9pbi5j
IGIvc3JjL2JhY2tlbmQvZXhlY3V0b3Ivbm9kZUhhc2hqb2luLmMKaW5kZXggMmY3MTcwNjA0ZC4u
YmE1YzFkNGI0OSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvZXhlY3V0b3Ivbm9kZUhhc2hqb2lu
LmMKKysrIGIvc3JjL2JhY2tlbmQvZXhlY3V0b3Ivbm9kZUhhc2hqb2luLmMKQEAgLTE5NSwxMCAr
MTk1LDEwIEBAIHN0YXRpYyBUdXBsZVRhYmxlU2xvdCAqRXhlY0hhc2hKb2luT3V0ZXJHZXRUdXBs
ZShQbGFuU3RhdGUgKm91dGVyTm9kZSwKIHN0YXRpYyBUdXBsZVRhYmxlU2xvdCAqRXhlY1BhcmFs
bGVsSGFzaEpvaW5PdXRlckdldFR1cGxlKFBsYW5TdGF0ZSAqb3V0ZXJOb2RlLAogCQkJCQkJCQkJ
CQkJCQkgSGFzaEpvaW5TdGF0ZSAqaGpzdGF0ZSwKIAkJCQkJCQkJCQkJCQkJIHVpbnQzMiAqaGFz
aHZhbHVlKTsKLXN0YXRpYyBUdXBsZVRhYmxlU2xvdCAqRXhlY0hhc2hKb2luR2V0U2F2ZWRUdXBs
ZShIYXNoSm9pblN0YXRlICpoanN0YXRlLAotCQkJCQkJCQkJCQkJIEJ1ZkZpbGUgKmZpbGUsCi0J
CQkJCQkJCQkJCQkgdWludDMyICpoYXNodmFsdWUsCi0JCQkJCQkJCQkJCQkgVHVwbGVUYWJsZVNs
b3QgKnR1cGxlU2xvdCk7CitzdGF0aWMgTWluaW1hbFR1cGxlIEV4ZWNIYXNoSm9pbkdldFNhdmVk
VHVwbGUoSGFzaEpvaW5TdGF0ZSAqaGpzdGF0ZSwKKwkJCQkJCQkJCQkJICBCdWZGaWxlICpmaWxl
LAorCQkJCQkJCQkJCQkgIHVpbnQzMiAqaGFzaHZhbHVlLAorCQkJCQkJCQkJCQkgIFN0cmluZ0lu
Zm8gYnVmKTsKIHN0YXRpYyBib29sIEV4ZWNIYXNoSm9pbk5ld0JhdGNoKEhhc2hKb2luU3RhdGUg
Kmhqc3RhdGUpOwogc3RhdGljIGJvb2wgRXhlY1BhcmFsbGVsSGFzaEpvaW5OZXdCYXRjaChIYXNo
Sm9pblN0YXRlICpoanN0YXRlKTsKIHN0YXRpYyB2b2lkIEV4ZWNQYXJhbGxlbEhhc2hKb2luUGFy
dGl0aW9uT3V0ZXIoSGFzaEpvaW5TdGF0ZSAqaGpzdGF0ZSk7CkBAIC05MjUsNiArOTI1LDcgQEAg
RXhlY0luaXRIYXNoSm9pbihIYXNoSm9pbiAqbm9kZSwgRVN0YXRlICplc3RhdGUsIGludCBlZmxh
Z3MpCiAJICovCiAJaGpzdGF0ZS0+aGpfSGFzaFRhYmxlID0gTlVMTDsKIAloanN0YXRlLT5oal9G
aXJzdE91dGVyVHVwbGVTbG90ID0gTlVMTDsKKwloanN0YXRlLT5oal9vdXRlclR1cGxlQnVmZmVy
ID0gTlVMTDsKIAogCWhqc3RhdGUtPmhqX0N1ckhhc2hWYWx1ZSA9IDA7CiAJaGpzdGF0ZS0+aGpf
Q3VyQnVja2V0Tm8gPSAwOwpAQCAtMTAzMCw2ICsxMDMxLDcgQEAgRXhlY0hhc2hKb2luT3V0ZXJH
ZXRUdXBsZShQbGFuU3RhdGUgKm91dGVyTm9kZSwKIAl9CiAJZWxzZSBpZiAoY3VyYmF0Y2ggPCBo
YXNodGFibGUtPm5iYXRjaCkKIAl7CisJCU1pbmltYWxUdXBsZSBtdHVwOwogCQlCdWZGaWxlICAg
ICpmaWxlID0gaGFzaHRhYmxlLT5vdXRlckJhdGNoRmlsZVtjdXJiYXRjaF07CiAKIAkJLyoKQEAg
LTEwMzksMTIgKzEwNDEsMzggQEAgRXhlY0hhc2hKb2luT3V0ZXJHZXRUdXBsZShQbGFuU3RhdGUg
Km91dGVyTm9kZSwKIAkJaWYgKGZpbGUgPT0gTlVMTCkKIAkJCXJldHVybiBOVUxMOwogCi0JCXNs
b3QgPSBFeGVjSGFzaEpvaW5HZXRTYXZlZFR1cGxlKGhqc3RhdGUsCisJCWlmICh1bmxpa2VseSho
anN0YXRlLT5oal9vdXRlclR1cGxlQnVmZmVyID09IE5VTEwpKQorCQl7CisJCQkvKgorCQkJICog
QXZvaWQgcmVhbGxvYyBtZW1vcnkgZm9yIE1pbmltYWxUdXBsZSwgd2UgYWxsb2MgYSBidWZmZXIK
KwkJCSAqIHRvIHJldXNlIHN0b3JlIE1pbmltYWxUdXBsZS4gTWVtb3J5Q29udGV4dCBzYW1lIGFz
IGhqc3RhdGUuCisJCQkgKi8KKwkJCU1lbW9yeUNvbnRleHQgb2xkY29udGV4dCA9IE1lbW9yeUNv
bnRleHRTd2l0Y2hUbyhHZXRNZW1vcnlDaHVua0NvbnRleHQoaGpzdGF0ZSkpOworCQkJaGpzdGF0
ZS0+aGpfb3V0ZXJUdXBsZUJ1ZmZlciA9IG1ha2VTdHJpbmdJbmZvKCk7CisJCQlNZW1vcnlDb250
ZXh0U3dpdGNoVG8ob2xkY29udGV4dCk7CisJCX0KKworCQltdHVwID0gRXhlY0hhc2hKb2luR2V0
U2F2ZWRUdXBsZShoanN0YXRlLAogCQkJCQkJCQkJCSBmaWxlLAogCQkJCQkJCQkJCSBoYXNodmFs
dWUsCi0JCQkJCQkJCQkJIGhqc3RhdGUtPmhqX091dGVyVHVwbGVTbG90KTsKLQkJaWYgKCFUdXBJ
c051bGwoc2xvdCkpCisJCQkJCQkJCQkJIGhqc3RhdGUtPmhqX291dGVyVHVwbGVCdWZmZXIpOwor
CQlpZiAobGlrZWx5KG10dXAgIT0gTlVMTCkpCisJCXsKKwkJCXNsb3QgPSBoanN0YXRlLT5oal9P
dXRlclR1cGxlU2xvdDsKKworCQkJLyoKKwkJCSAqIG10dXAgaXMgaG9sZCBpbiBoanN0YXRlLT5o
al9vdXRlclR1cGxlQnVmZmVyLCBzbyB3ZSBjYW4gdXNpbmcKKwkJCSAqIHNob3VsZEZyZWUgYXMg
ZmFsc2UgdG8gY2FsbCBFeGVjRm9yY2VTdG9yZU1pbmltYWxUdXBsZSgpLgorCQkJICogCisJCQkg
KiBXaGVuIHNsb3QgaXMgVFRTT3BzTWluaW1hbFR1cGxlIHdlIGNhbiBhdm9pZCByZWFsbG9jIG1l
bW9yeSBmb3IKKwkJCSAqIG5ldyBNaW5pbWFsVHVwbGUocmV1c2UgU3RyaW5nSW5mbyB0byBjYWxs
IEV4ZWNIYXNoSm9pbkdldFNhdmVkVHVwbGUpLgorCQkJICogCisJCQkgKiBNb3JlIGltcG9ydGFu
dGx5LCBpbiBub24tVFRTT3BzTWluaW1hbFR1cGxlIHNjZW5hcmlvcywgaXQgY2FuIGF2b2lkCisJ
CQkgKiByZWZvcm0obWF0ZXJpYWxpemUpIHR1cGxlKHNlZSBFeGVjRm9yY2VTdG9yZU1pbmltYWxU
dXBsZSkuCisJCQkgKi8KKwkJCUV4ZWNGb3JjZVN0b3JlTWluaW1hbFR1cGxlKG10dXAsIHNsb3Qs
IGZhbHNlKTsKIAkJCXJldHVybiBzbG90OworCQl9CiAJfQogCiAJLyogRW5kIG9mIHRoaXMgYmF0
Y2ggKi8KQEAgLTExMzMsNyArMTE2MSw2IEBAIEV4ZWNIYXNoSm9pbk5ld0JhdGNoKEhhc2hKb2lu
U3RhdGUgKmhqc3RhdGUpCiAJaW50CQkJbmJhdGNoOwogCWludAkJCWN1cmJhdGNoOwogCUJ1ZkZp
bGUgICAgKmlubmVyRmlsZTsKLQlUdXBsZVRhYmxlU2xvdCAqc2xvdDsKIAl1aW50MzIJCWhhc2h2
YWx1ZTsKIAogCW5iYXRjaCA9IGhhc2h0YWJsZS0+bmJhdGNoOwpAQCAtMTIyNCwyMSArMTI1MSwy
NSBAQCBFeGVjSGFzaEpvaW5OZXdCYXRjaChIYXNoSm9pblN0YXRlICpoanN0YXRlKQogCiAJaWYg
KGlubmVyRmlsZSAhPSBOVUxMKQogCXsKKwkJU3RyaW5nSW5mb0RhdGEgYnVmOworCQlNaW5pbWFs
VHVwbGUgdHVwbGU7CisKIAkJaWYgKEJ1ZkZpbGVTZWVrKGlubmVyRmlsZSwgMCwgMCwgU0VFS19T
RVQpKQogCQkJZXJlcG9ydChFUlJPUiwKIAkJCQkJKGVycmNvZGVfZm9yX2ZpbGVfYWNjZXNzKCks
CiAJCQkJCSBlcnJtc2coImNvdWxkIG5vdCByZXdpbmQgaGFzaC1qb2luIHRlbXBvcmFyeSBmaWxl
IikpKTsKIAotCQl3aGlsZSAoKHNsb3QgPSBFeGVjSGFzaEpvaW5HZXRTYXZlZFR1cGxlKGhqc3Rh
dGUsCi0JCQkJCQkJCQkJCQkgaW5uZXJGaWxlLAotCQkJCQkJCQkJCQkJICZoYXNodmFsdWUsCi0J
CQkJCQkJCQkJCQkgaGpzdGF0ZS0+aGpfSGFzaFR1cGxlU2xvdCkpKQorCQlpbml0U3RyaW5nSW5m
bygmYnVmKTsKKwkJd2hpbGUgKCh0dXBsZSA9IEV4ZWNIYXNoSm9pbkdldFNhdmVkVHVwbGUoaGpz
dGF0ZSwKKwkJCQkJCQkJCQkJCSAgaW5uZXJGaWxlLAorCQkJCQkJCQkJCQkJICAmaGFzaHZhbHVl
LAorCQkJCQkJCQkJCQkJICAmYnVmKSkpCiAJCXsKIAkJCS8qCiAJCQkgKiBOT1RFOiBzb21lIHR1
cGxlcyBtYXkgYmUgc2VudCB0byBmdXR1cmUgYmF0Y2hlcy4gIEFsc28sIGl0IGlzCiAJCQkgKiBw
b3NzaWJsZSBmb3IgaGFzaHRhYmxlLT5uYmF0Y2ggdG8gYmUgaW5jcmVhc2VkIGhlcmUhCiAJCQkg
Ki8KLQkJCUV4ZWNIYXNoVGFibGVJbnNlcnQoaGFzaHRhYmxlLCBzbG90LCBoYXNodmFsdWUpOwor
CQkJRXhlY0hhc2hUYWJsZUluc2VydFR1cGxlKGhhc2h0YWJsZSwgdHVwbGUsIGhhc2h2YWx1ZSk7
CiAJCX0KIAogCQkvKgpAQCAtMTI0Nyw2ICsxMjc4LDcgQEAgRXhlY0hhc2hKb2luTmV3QmF0Y2go
SGFzaEpvaW5TdGF0ZSAqaGpzdGF0ZSkKIAkJICovCiAJCUJ1ZkZpbGVDbG9zZShpbm5lckZpbGUp
OwogCQloYXNodGFibGUtPmlubmVyQmF0Y2hGaWxlW2N1cmJhdGNoXSA9IE5VTEw7CisJCXBmcmVl
KGJ1Zi5kYXRhKTsKIAl9CiAKIAkvKgpAQCAtMTI5Nyw3ICsxMzI5LDYgQEAgRXhlY1BhcmFsbGVs
SGFzaEpvaW5OZXdCYXRjaChIYXNoSm9pblN0YXRlICpoanN0YXRlKQogCXsKIAkJdWludDMyCQlo
YXNodmFsdWU7CiAJCU1pbmltYWxUdXBsZSB0dXBsZTsKLQkJVHVwbGVUYWJsZVNsb3QgKnNsb3Q7
CiAKIAkJaWYgKCFoYXNodGFibGUtPmJhdGNoZXNbYmF0Y2hub10uZG9uZSkKIAkJewpAQCAtMTMy
OSwxMiArMTM2MCw5IEBAIEV4ZWNQYXJhbGxlbEhhc2hKb2luTmV3QmF0Y2goSGFzaEpvaW5TdGF0
ZSAqaGpzdGF0ZSkKIAkJCQkJd2hpbGUgKCh0dXBsZSA9IHN0c19wYXJhbGxlbF9zY2FuX25leHQo
aW5uZXJfdHVwbGVzLAogCQkJCQkJCQkJCQkJCQkgICAmaGFzaHZhbHVlKSkpCiAJCQkJCXsKLQkJ
CQkJCUV4ZWNGb3JjZVN0b3JlTWluaW1hbFR1cGxlKHR1cGxlLAotCQkJCQkJCQkJCQkJICAgaGpz
dGF0ZS0+aGpfSGFzaFR1cGxlU2xvdCwKLQkJCQkJCQkJCQkJCSAgIGZhbHNlKTsKLQkJCQkJCXNs
b3QgPSBoanN0YXRlLT5oal9IYXNoVHVwbGVTbG90OwotCQkJCQkJRXhlY1BhcmFsbGVsSGFzaFRh
YmxlSW5zZXJ0Q3VycmVudEJhdGNoKGhhc2h0YWJsZSwgc2xvdCwKLQkJCQkJCQkJCQkJCQkJCQlo
YXNodmFsdWUpOworCQkJCQkJRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0VHVwbGVDdXJyZW50
QmF0Y2goaGFzaHRhYmxlLAorCQkJCQkJCQkJCQkJCQkJCQkgdHVwbGUsCisJCQkJCQkJCQkJCQkJ
CQkJCSBoYXNodmFsdWUpOwogCQkJCQl9CiAJCQkJCXN0c19lbmRfcGFyYWxsZWxfc2Nhbihpbm5l
cl90dXBsZXMpOwogCQkJCQlCYXJyaWVyQXJyaXZlQW5kV2FpdChiYXRjaF9iYXJyaWVyLApAQCAt
MTQ0OCwxNCArMTQ3NiwxNCBAQCBFeGVjSGFzaEpvaW5TYXZlVHVwbGUoTWluaW1hbFR1cGxlIHR1
cGxlLCB1aW50MzIgaGFzaHZhbHVlLAogICogRXhlY0hhc2hKb2luR2V0U2F2ZWRUdXBsZQogICoJ
CXJlYWQgdGhlIG5leHQgdHVwbGUgZnJvbSBhIGJhdGNoIGZpbGUuICBSZXR1cm4gTlVMTCBpZiBu
byBtb3JlLgogICoKLSAqIE9uIHN1Y2Nlc3MsICpoYXNodmFsdWUgaXMgc2V0IHRvIHRoZSB0dXBs
ZSdzIGhhc2ggdmFsdWUsIGFuZCB0aGUgdHVwbGUKLSAqIGl0c2VsZiBpcyBzdG9yZWQgaW4gdGhl
IGdpdmVuIHNsb3QuCisgKiBPbiBzdWNjZXNzLCAqaGFzaHZhbHVlIGlzIHNldCB0byB0aGUgdHVw
bGUncyBoYXNoIHZhbHVlLCBhbmQgcmV0dXJuCisgKiB0aGUgdHVwbGUoc3RvcmVkIGluIHRoZSBn
aXZlbiBidWYpIGl0c2VsZi4KICAqLwotc3RhdGljIFR1cGxlVGFibGVTbG90ICoKK3N0YXRpYyBN
aW5pbWFsVHVwbGUKIEV4ZWNIYXNoSm9pbkdldFNhdmVkVHVwbGUoSGFzaEpvaW5TdGF0ZSAqaGpz
dGF0ZSwKIAkJCQkJCSAgQnVmRmlsZSAqZmlsZSwKIAkJCQkJCSAgdWludDMyICpoYXNodmFsdWUs
Ci0JCQkJCQkgIFR1cGxlVGFibGVTbG90ICp0dXBsZVNsb3QpCisJCQkJCQkgIFN0cmluZ0luZm8g
YnVmKQogewogCXVpbnQzMgkJaGVhZGVyWzJdOwogCXNpemVfdAkJbnJlYWQ7CkBAIC0xNDc0LDE5
ICsxNTAyLDIwIEBAIEV4ZWNIYXNoSm9pbkdldFNhdmVkVHVwbGUoSGFzaEpvaW5TdGF0ZSAqaGpz
dGF0ZSwKIAkgKiBjaGVhdGluZy4KIAkgKi8KIAlucmVhZCA9IEJ1ZkZpbGVSZWFkTWF5YmVFT0Yo
ZmlsZSwgaGVhZGVyLCBzaXplb2YoaGVhZGVyKSwgdHJ1ZSk7Ci0JaWYgKG5yZWFkID09IDApCQkJ
CS8qIGVuZCBvZiBmaWxlICovCi0JewotCQlFeGVjQ2xlYXJUdXBsZSh0dXBsZVNsb3QpOworCWlm
ICh1bmxpa2VseShucmVhZCA9PSAwKSkJCQkJLyogZW5kIG9mIGZpbGUgKi8KIAkJcmV0dXJuIE5V
TEw7Ci0JfQorCisJYnVmLT5sZW4gPSAwOwkvKiBpbmxpbmUgcmVzZXRTdHJpbmdJbmZvKGJ1Zik7
ICovCisJZW5sYXJnZVN0cmluZ0luZm8oYnVmLCBoZWFkZXJbMV0pOworCWJ1Zi0+bGVuID0gaGVh
ZGVyWzFdOwogCSpoYXNodmFsdWUgPSBoZWFkZXJbMF07Ci0JdHVwbGUgPSAoTWluaW1hbFR1cGxl
KSBwYWxsb2MoaGVhZGVyWzFdKTsKKwl0dXBsZSA9IChNaW5pbWFsVHVwbGUpIGJ1Zi0+ZGF0YTsK
IAl0dXBsZS0+dF9sZW4gPSBoZWFkZXJbMV07CiAJQnVmRmlsZVJlYWRFeGFjdChmaWxlLAogCQkJ
CQkgKGNoYXIgKikgdHVwbGUgKyBzaXplb2YodWludDMyKSwKIAkJCQkJIGhlYWRlclsxXSAtIHNp
emVvZih1aW50MzIpKTsKLQlFeGVjRm9yY2VTdG9yZU1pbmltYWxUdXBsZSh0dXBsZSwgdHVwbGVT
bG90LCB0cnVlKTsKLQlyZXR1cm4gdHVwbGVTbG90OworCisJcmV0dXJuIHR1cGxlOwogfQogCiAK
ZGlmZiAtLWdpdCBhL3NyYy9pbmNsdWRlL2V4ZWN1dG9yL25vZGVIYXNoLmggYi9zcmMvaW5jbHVk
ZS9leGVjdXRvci9ub2RlSGFzaC5oCmluZGV4IGU0ZWI3YmM2MzUuLjZjYzIyNjQ2OTIgMTAwNjQ0
Ci0tLSBhL3NyYy9pbmNsdWRlL2V4ZWN1dG9yL25vZGVIYXNoLmgKKysrIGIvc3JjL2luY2x1ZGUv
ZXhlY3V0b3Ivbm9kZUhhc2guaApAQCAtMzMsMTUgKzMzLDIxIEBAIGV4dGVybiB2b2lkIEV4ZWNI
YXNoVGFibGVEZXRhY2hCYXRjaChIYXNoSm9pblRhYmxlIGhhc2h0YWJsZSk7CiBleHRlcm4gdm9p
ZCBFeGVjUGFyYWxsZWxIYXNoVGFibGVTZXRDdXJyZW50QmF0Y2goSGFzaEpvaW5UYWJsZSBoYXNo
dGFibGUsCiAJCQkJCQkJCQkJCQkgaW50IGJhdGNobm8pOwogCi1leHRlcm4gdm9pZCBFeGVjSGFz
aFRhYmxlSW5zZXJ0KEhhc2hKb2luVGFibGUgaGFzaHRhYmxlLAotCQkJCQkJCQlUdXBsZVRhYmxl
U2xvdCAqc2xvdCwKLQkJCQkJCQkJdWludDMyIGhhc2h2YWx1ZSk7CitleHRlcm4gdm9pZCBFeGVj
SGFzaFRhYmxlSW5zZXJ0U2xvdChIYXNoSm9pblRhYmxlIGhhc2h0YWJsZSwKKwkJCQkJCQkJCVR1
cGxlVGFibGVTbG90ICpzbG90LAorCQkJCQkJCQkJdWludDMyIGhhc2h2YWx1ZSk7CitleHRlcm4g
dm9pZCBFeGVjSGFzaFRhYmxlSW5zZXJ0VHVwbGUoSGFzaEpvaW5UYWJsZSBoYXNodGFibGUsCisJ
CQkJCQkJCQkgTWluaW1hbFR1cGxlIHR1cGxlLAorCQkJCQkJCQkJIHVpbnQzMiBoYXNodmFsdWUp
OwogZXh0ZXJuIHZvaWQgRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0KEhhc2hKb2luVGFibGUg
aGFzaHRhYmxlLAogCQkJCQkJCQkJCVR1cGxlVGFibGVTbG90ICpzbG90LAogCQkJCQkJCQkJCXVp
bnQzMiBoYXNodmFsdWUpOwotZXh0ZXJuIHZvaWQgRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0
Q3VycmVudEJhdGNoKEhhc2hKb2luVGFibGUgaGFzaHRhYmxlLAotCQkJCQkJCQkJCQkJCVR1cGxl
VGFibGVTbG90ICpzbG90LAotCQkJCQkJCQkJCQkJCXVpbnQzMiBoYXNodmFsdWUpOworZXh0ZXJu
IHZvaWQgRXhlY1BhcmFsbGVsSGFzaFRhYmxlSW5zZXJ0U2xvdEN1cnJlbnRCYXRjaChIYXNoSm9p
blRhYmxlIGhhc2h0YWJsZSwKKwkJCQkJCQkJCQkJCQkJVHVwbGVUYWJsZVNsb3QgKnNsb3QsCisJ
CQkJCQkJCQkJCQkJCXVpbnQzMiBoYXNodmFsdWUpOworZXh0ZXJuIHZvaWQgRXhlY1BhcmFsbGVs
SGFzaFRhYmxlSW5zZXJ0VHVwbGVDdXJyZW50QmF0Y2goSGFzaEpvaW5UYWJsZSBoYXNodGFibGUs
CisJCQkJCQkJCQkJCQkJCSBNaW5pbWFsVHVwbGUgdHVwbGUsCisJCQkJCQkJCQkJCQkJCSB1aW50
MzIgaGFzaHZhbHVlKTsKIGV4dGVybiB2b2lkIEV4ZWNIYXNoR2V0QnVja2V0QW5kQmF0Y2goSGFz
aEpvaW5UYWJsZSBoYXNodGFibGUsCiAJCQkJCQkJCQkgIHVpbnQzMiBoYXNodmFsdWUsCiAJCQkJ
CQkJCQkgIGludCAqYnVja2V0bm8sCmRpZmYgLS1naXQgYS9zcmMvaW5jbHVkZS9ub2Rlcy9leGVj
bm9kZXMuaCBiL3NyYy9pbmNsdWRlL25vZGVzL2V4ZWNub2Rlcy5oCmluZGV4IGFmN2Q4ZmQxZTcu
LjI5OWY2NmFmMTMgMTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL25vZGVzL2V4ZWNub2Rlcy5oCisr
KyBiL3NyYy9pbmNsdWRlL25vZGVzL2V4ZWNub2Rlcy5oCkBAIC0yMjI1LDYgKzIyMjUsNyBAQCB0
eXBlZGVmIHN0cnVjdCBIYXNoSm9pblN0YXRlCiAJVHVwbGVUYWJsZVNsb3QgKmhqX051bGxPdXRl
clR1cGxlU2xvdDsKIAlUdXBsZVRhYmxlU2xvdCAqaGpfTnVsbElubmVyVHVwbGVTbG90OwogCVR1
cGxlVGFibGVTbG90ICpoal9GaXJzdE91dGVyVHVwbGVTbG90OworCVN0cmluZ0luZm8JaGpfb3V0
ZXJUdXBsZUJ1ZmZlcjsKIAlpbnQJCQloal9Kb2luU3RhdGU7CiAJYm9vbAkJaGpfTWF0Y2hlZE91
dGVyOwogCWJvb2wJCWhqX091dGVyTm90RW1wdHk7Cg==
------=_Part_35779_1338760973.1724328493004--
Attachments:
[application/octet-stream] optimize-hashjoin-master.patch (11.9K, ../../[email protected]/2-optimize-hashjoin-master.patch)
download | inline diff:
diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c
index 570a90ebe1..5e702a37ff 100644
--- a/src/backend/executor/nodeHash.c
+++ b/src/backend/executor/nodeHash.c
@@ -190,7 +190,7 @@ MultiExecPrivateHash(HashState *node)
else
{
/* Not subject to skew optimization, so insert normally */
- ExecHashTableInsert(hashtable, slot, hashvalue);
+ ExecHashTableInsertSlot(hashtable, slot, hashvalue);
}
hashtable->totalTuples += 1;
}
@@ -1594,7 +1594,7 @@ ExecParallelHashIncreaseNumBuckets(HashJoinTable hashtable)
}
/*
- * ExecHashTableInsert
+ * ExecHashTableInsertSlot
* insert a tuple into the hash table depending on the hash value
* it may just go to a temp file for later batches
*
@@ -1605,12 +1605,29 @@ ExecParallelHashIncreaseNumBuckets(HashJoinTable hashtable)
* worth the messiness required.
*/
void
-ExecHashTableInsert(HashJoinTable hashtable,
- TupleTableSlot *slot,
- uint32 hashvalue)
+ExecHashTableInsertSlot(HashJoinTable hashtable,
+ TupleTableSlot *slot,
+ uint32 hashvalue)
{
bool shouldFree;
MinimalTuple tuple = ExecFetchSlotMinimalTuple(slot, &shouldFree);
+
+ ExecHashTableInsertTuple(hashtable, tuple, hashvalue);
+
+ if (shouldFree)
+ heap_free_minimal_tuple(tuple);
+}
+
+/*
+ * ExecHashTableInsertTuple
+ * insert a tuple into the hash table depending on the hash value
+ * it may just go to a temp file for later batches
+ */
+void
+ExecHashTableInsertTuple(HashJoinTable hashtable,
+ MinimalTuple tuple,
+ uint32 hashvalue)
+{
int bucketno;
int batchno;
@@ -1685,9 +1702,6 @@ ExecHashTableInsert(HashJoinTable hashtable,
&hashtable->innerBatchFile[batchno],
hashtable);
}
-
- if (shouldFree)
- heap_free_minimal_tuple(tuple);
}
/*
@@ -1761,12 +1775,10 @@ retry:
* tuples that belong in the current batch once growth has been disabled.
*/
void
-ExecParallelHashTableInsertCurrentBatch(HashJoinTable hashtable,
- TupleTableSlot *slot,
- uint32 hashvalue)
+ExecParallelHashTableInsertTupleCurrentBatch(HashJoinTable hashtable,
+ MinimalTuple tuple,
+ uint32 hashvalue)
{
- bool shouldFree;
- MinimalTuple tuple = ExecFetchSlotMinimalTuple(slot, &shouldFree);
HashJoinTuple hashTuple;
dsa_pointer shared;
int batchno;
@@ -1782,6 +1794,21 @@ ExecParallelHashTableInsertCurrentBatch(HashJoinTable hashtable,
HeapTupleHeaderClearMatch(HJTUPLE_MINTUPLE(hashTuple));
ExecParallelHashPushTuple(&hashtable->buckets.shared[bucketno],
hashTuple, shared);
+}
+
+/*
+ * like ExecParallelHashTableInsertTupleCurrentBatch,
+ * but this function accept a TupleTableSlot
+ */
+void
+ExecParallelHashTableInsertSlotCurrentBatch(HashJoinTable hashtable,
+ TupleTableSlot *slot,
+ uint32 hashvalue)
+{
+ bool shouldFree;
+ MinimalTuple tuple = ExecFetchSlotMinimalTuple(slot, &shouldFree);
+
+ ExecParallelHashTableInsertTupleCurrentBatch(hashtable, tuple, hashvalue);
if (shouldFree)
heap_free_minimal_tuple(tuple);
@@ -2454,7 +2481,7 @@ ExecHashGetSkewBucket(HashJoinTable hashtable, uint32 hashvalue)
* Insert a tuple into the skew hashtable.
*
* This should generally match up with the current-batch case in
- * ExecHashTableInsert.
+ * ExecHashTableInsertSlot.
*/
static void
ExecHashSkewTableInsert(HashJoinTable hashtable,
@@ -2534,8 +2561,8 @@ ExecHashRemoveNextSkewBucket(HashJoinTable hashtable)
Size tupleSize;
/*
- * This code must agree with ExecHashTableInsert. We do not use
- * ExecHashTableInsert directly as ExecHashTableInsert expects a
+ * This code must agree with ExecHashTableInsertSlot. We do not use
+ * ExecHashTableInsertSlot directly as ExecHashTableInsertSlot expects a
* TupleTableSlot while we already have HashJoinTuples.
*/
tuple = HJTUPLE_MINTUPLE(hashTuple);
diff --git a/src/backend/executor/nodeHashjoin.c b/src/backend/executor/nodeHashjoin.c
index 2f7170604d..ba5c1d4b49 100644
--- a/src/backend/executor/nodeHashjoin.c
+++ b/src/backend/executor/nodeHashjoin.c
@@ -195,10 +195,10 @@ static TupleTableSlot *ExecHashJoinOuterGetTuple(PlanState *outerNode,
static TupleTableSlot *ExecParallelHashJoinOuterGetTuple(PlanState *outerNode,
HashJoinState *hjstate,
uint32 *hashvalue);
-static TupleTableSlot *ExecHashJoinGetSavedTuple(HashJoinState *hjstate,
- BufFile *file,
- uint32 *hashvalue,
- TupleTableSlot *tupleSlot);
+static MinimalTuple ExecHashJoinGetSavedTuple(HashJoinState *hjstate,
+ BufFile *file,
+ uint32 *hashvalue,
+ StringInfo buf);
static bool ExecHashJoinNewBatch(HashJoinState *hjstate);
static bool ExecParallelHashJoinNewBatch(HashJoinState *hjstate);
static void ExecParallelHashJoinPartitionOuter(HashJoinState *hjstate);
@@ -925,6 +925,7 @@ ExecInitHashJoin(HashJoin *node, EState *estate, int eflags)
*/
hjstate->hj_HashTable = NULL;
hjstate->hj_FirstOuterTupleSlot = NULL;
+ hjstate->hj_outerTupleBuffer = NULL;
hjstate->hj_CurHashValue = 0;
hjstate->hj_CurBucketNo = 0;
@@ -1030,6 +1031,7 @@ ExecHashJoinOuterGetTuple(PlanState *outerNode,
}
else if (curbatch < hashtable->nbatch)
{
+ MinimalTuple mtup;
BufFile *file = hashtable->outerBatchFile[curbatch];
/*
@@ -1039,12 +1041,38 @@ ExecHashJoinOuterGetTuple(PlanState *outerNode,
if (file == NULL)
return NULL;
- slot = ExecHashJoinGetSavedTuple(hjstate,
+ if (unlikely(hjstate->hj_outerTupleBuffer == NULL))
+ {
+ /*
+ * Avoid realloc memory for MinimalTuple, we alloc a buffer
+ * to reuse store MinimalTuple. MemoryContext same as hjstate.
+ */
+ MemoryContext oldcontext = MemoryContextSwitchTo(GetMemoryChunkContext(hjstate));
+ hjstate->hj_outerTupleBuffer = makeStringInfo();
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ mtup = ExecHashJoinGetSavedTuple(hjstate,
file,
hashvalue,
- hjstate->hj_OuterTupleSlot);
- if (!TupIsNull(slot))
+ hjstate->hj_outerTupleBuffer);
+ if (likely(mtup != NULL))
+ {
+ slot = hjstate->hj_OuterTupleSlot;
+
+ /*
+ * mtup is hold in hjstate->hj_outerTupleBuffer, so we can using
+ * shouldFree as false to call ExecForceStoreMinimalTuple().
+ *
+ * When slot is TTSOpsMinimalTuple we can avoid realloc memory for
+ * new MinimalTuple(reuse StringInfo to call ExecHashJoinGetSavedTuple).
+ *
+ * More importantly, in non-TTSOpsMinimalTuple scenarios, it can avoid
+ * reform(materialize) tuple(see ExecForceStoreMinimalTuple).
+ */
+ ExecForceStoreMinimalTuple(mtup, slot, false);
return slot;
+ }
}
/* End of this batch */
@@ -1133,7 +1161,6 @@ ExecHashJoinNewBatch(HashJoinState *hjstate)
int nbatch;
int curbatch;
BufFile *innerFile;
- TupleTableSlot *slot;
uint32 hashvalue;
nbatch = hashtable->nbatch;
@@ -1224,21 +1251,25 @@ ExecHashJoinNewBatch(HashJoinState *hjstate)
if (innerFile != NULL)
{
+ StringInfoData buf;
+ MinimalTuple tuple;
+
if (BufFileSeek(innerFile, 0, 0, SEEK_SET))
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not rewind hash-join temporary file")));
- while ((slot = ExecHashJoinGetSavedTuple(hjstate,
- innerFile,
- &hashvalue,
- hjstate->hj_HashTupleSlot)))
+ initStringInfo(&buf);
+ while ((tuple = ExecHashJoinGetSavedTuple(hjstate,
+ innerFile,
+ &hashvalue,
+ &buf)))
{
/*
* NOTE: some tuples may be sent to future batches. Also, it is
* possible for hashtable->nbatch to be increased here!
*/
- ExecHashTableInsert(hashtable, slot, hashvalue);
+ ExecHashTableInsertTuple(hashtable, tuple, hashvalue);
}
/*
@@ -1247,6 +1278,7 @@ ExecHashJoinNewBatch(HashJoinState *hjstate)
*/
BufFileClose(innerFile);
hashtable->innerBatchFile[curbatch] = NULL;
+ pfree(buf.data);
}
/*
@@ -1297,7 +1329,6 @@ ExecParallelHashJoinNewBatch(HashJoinState *hjstate)
{
uint32 hashvalue;
MinimalTuple tuple;
- TupleTableSlot *slot;
if (!hashtable->batches[batchno].done)
{
@@ -1329,12 +1360,9 @@ ExecParallelHashJoinNewBatch(HashJoinState *hjstate)
while ((tuple = sts_parallel_scan_next(inner_tuples,
&hashvalue)))
{
- ExecForceStoreMinimalTuple(tuple,
- hjstate->hj_HashTupleSlot,
- false);
- slot = hjstate->hj_HashTupleSlot;
- ExecParallelHashTableInsertCurrentBatch(hashtable, slot,
- hashvalue);
+ ExecParallelHashTableInsertTupleCurrentBatch(hashtable,
+ tuple,
+ hashvalue);
}
sts_end_parallel_scan(inner_tuples);
BarrierArriveAndWait(batch_barrier,
@@ -1448,14 +1476,14 @@ ExecHashJoinSaveTuple(MinimalTuple tuple, uint32 hashvalue,
* ExecHashJoinGetSavedTuple
* read the next tuple from a batch file. Return NULL if no more.
*
- * On success, *hashvalue is set to the tuple's hash value, and the tuple
- * itself is stored in the given slot.
+ * On success, *hashvalue is set to the tuple's hash value, and return
+ * the tuple(stored in the given buf) itself.
*/
-static TupleTableSlot *
+static MinimalTuple
ExecHashJoinGetSavedTuple(HashJoinState *hjstate,
BufFile *file,
uint32 *hashvalue,
- TupleTableSlot *tupleSlot)
+ StringInfo buf)
{
uint32 header[2];
size_t nread;
@@ -1474,19 +1502,20 @@ ExecHashJoinGetSavedTuple(HashJoinState *hjstate,
* cheating.
*/
nread = BufFileReadMaybeEOF(file, header, sizeof(header), true);
- if (nread == 0) /* end of file */
- {
- ExecClearTuple(tupleSlot);
+ if (unlikely(nread == 0)) /* end of file */
return NULL;
- }
+
+ buf->len = 0; /* inline resetStringInfo(buf); */
+ enlargeStringInfo(buf, header[1]);
+ buf->len = header[1];
*hashvalue = header[0];
- tuple = (MinimalTuple) palloc(header[1]);
+ tuple = (MinimalTuple) buf->data;
tuple->t_len = header[1];
BufFileReadExact(file,
(char *) tuple + sizeof(uint32),
header[1] - sizeof(uint32));
- ExecForceStoreMinimalTuple(tuple, tupleSlot, true);
- return tupleSlot;
+
+ return tuple;
}
diff --git a/src/include/executor/nodeHash.h b/src/include/executor/nodeHash.h
index e4eb7bc635..6cc2264692 100644
--- a/src/include/executor/nodeHash.h
+++ b/src/include/executor/nodeHash.h
@@ -33,15 +33,21 @@ extern void ExecHashTableDetachBatch(HashJoinTable hashtable);
extern void ExecParallelHashTableSetCurrentBatch(HashJoinTable hashtable,
int batchno);
-extern void ExecHashTableInsert(HashJoinTable hashtable,
- TupleTableSlot *slot,
- uint32 hashvalue);
+extern void ExecHashTableInsertSlot(HashJoinTable hashtable,
+ TupleTableSlot *slot,
+ uint32 hashvalue);
+extern void ExecHashTableInsertTuple(HashJoinTable hashtable,
+ MinimalTuple tuple,
+ uint32 hashvalue);
extern void ExecParallelHashTableInsert(HashJoinTable hashtable,
TupleTableSlot *slot,
uint32 hashvalue);
-extern void ExecParallelHashTableInsertCurrentBatch(HashJoinTable hashtable,
- TupleTableSlot *slot,
- uint32 hashvalue);
+extern void ExecParallelHashTableInsertSlotCurrentBatch(HashJoinTable hashtable,
+ TupleTableSlot *slot,
+ uint32 hashvalue);
+extern void ExecParallelHashTableInsertTupleCurrentBatch(HashJoinTable hashtable,
+ MinimalTuple tuple,
+ uint32 hashvalue);
extern void ExecHashGetBucketAndBatch(HashJoinTable hashtable,
uint32 hashvalue,
int *bucketno,
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index af7d8fd1e7..299f66af13 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2225,6 +2225,7 @@ typedef struct HashJoinState
TupleTableSlot *hj_NullOuterTupleSlot;
TupleTableSlot *hj_NullInnerTupleSlot;
TupleTableSlot *hj_FirstOuterTupleSlot;
+ StringInfo hj_outerTupleBuffer;
int hj_JoinState;
bool hj_MatchedOuter;
bool hj_OuterNotEmpty;
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: optimize hashjoin
2024-08-22 12:08 Re: optimize hashjoin bucoo <[email protected]>
@ 2024-08-22 12:14 ` Kirill Reshke <[email protected]>
1 sibling, 0 replies; 80+ messages in thread
From: Kirill Reshke @ 2024-08-22 12:14 UTC (permalink / raw)
To: bucoo <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers; Tom Lane <[email protected]>; Robert Haas <[email protected]>
On Thu, 22 Aug 2024 at 17:08, bucoo <[email protected]> wrote:
>
> > 0) The patch does not apply anymore, thanks to David committing a patch
>
> > yesterday. Attached is a patch rebased on top of current master.
>
> That patch is based on PG17. I have now rewritten it based on the master
> branch and added some comments.
>
>
> > 1) Wouldn't it be easier (and just as efficient) to use slots with
>
> > TTSOpsMinimalTuple, i.e. backed by a minimal tuple?
>
> Use diffent kind of slot, the ExecEvalExpr function will report an error.
>
>
> > 2) I find the naming of the new functions a bit confusing. We now have
> > the "original" functions working with slots, and then also functions
> > with "Tuple" working with tuples. Seems asymmetric.
>
> In net patch function name renamed to ExecHashTableInsertSlot and
> ExecHashTableInsertTuple,
>
> also ExecParallelHashTableInsertSlotCurrentBatch and
> ExecParallelHashTableInsertTupleCurrentBatch.
>
>
> > 3) The code in ExecHashJoinOuterGetTuple is hard to understand, it'd
> > very much benefit from some comments. I'm a bit unsure if the likely()
> > and unlikely() hints really help.
>
> In new patch added some comments.
>
> "Likely" and "unlikely" might offer only marginal help on some CPUs and
> might not be beneficial at all on other platforms (I think).
>
>
> > 4) Is the hj_outerTupleBuffer buffer really needed / effective? I'd bet
> > just using palloc() will work just as well, thanks to the AllocSet
> > caching etc.
>
> The hj_outerTupleBuffer avoid reform(materialize) tuple in
> non-TTSOpsMinimalTuple scenarios,
>
> see ExecForceStoreMinimalTuple. I added some comments in new patch.
>
>
> > Can you provide more information about the benchmark you did? What
> > hardware, what scale, PostgreSQL configuration, which of the 22 queries
> > are improved, etc.
> >
> > I ran TPC-H with 1GB and 10GB scales on two machines, and I see pretty
> > much no difference compared to master. However, it occurred to me the
> > patch only ever helps if we increase the number of batches during
> > execution, in which case we need to move tuples to the right batch.
>
> Only parallel HashJoin speed up to ~2x(all data cached in memory),
>
> not full query, include non-parallel HashJoin.
>
> non-parallel HashJoin only when batchs large then one will speed up,
>
> because this patch only optimize for read batchs tuples to memory.
>
> Hi
likely/unlikely usage can be justified via benchmark. Parallel HashJoin
speed up still also can be verified via benchmark. Either benchmark script
or benchmark result, and it is better to provide both.
--
Best regards,
Kirill Reshke
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: optimize hashjoin
2024-08-22 12:08 Re: optimize hashjoin bucoo <[email protected]>
@ 2024-08-22 19:26 ` Tomas Vondra <[email protected]>
2024-08-23 11:02 ` 答复: optimize hashjoin bucoo <[email protected]>
1 sibling, 1 reply; 80+ messages in thread
From: Tomas Vondra @ 2024-08-22 19:26 UTC (permalink / raw)
To: bucoo <[email protected]>; pgsql-hackers; +Cc: 'Tom Lane' <[email protected]>; 'Robert Haas' <[email protected]>
Hi,
It seems you responded by writing a new message and just copying the
subject, which unfortunately doesn't set the headers used for threading
(e.g. in archives). Please just respond to the message.
Or maybe your client does not set the References/In-Reply-To headers
correctly. Not sure which mail client you're using.
On 8/22/24 14:08, bucoo wrote:
>
>> 0) The patch does not apply anymore, thanks to David committing a patch
>
>> yesterday. Attached is a patch rebased on top of current master.
>
> That patch is based on PG17. I have now rewritten it based on the master
> branch and added some comments.
>
Thanks. Yeah, patches should be based on "master".
>
>> 1) Wouldn't it be easier (and just as efficient) to use slots with
>
>> TTSOpsMinimalTuple, i.e. backed by a minimal tuple?
>
> Use diffent kind of slot, the ExecEvalExpr function will report an error.
>
Hmm, I haven't tried so it's possible it wouldn't work.
>
>> 2) I find the naming of the new functions a bit confusing. We now have
>> the "original" functions working with slots, and then also functions
>> with "Tuple" working with tuples. Seems asymmetric.
>
> In net patch function name renamed to ExecHashTableInsertSlot and
> ExecHashTableInsertTuple,
>
> also ExecParallelHashTableInsertSlotCurrentBatch and
> ExecParallelHashTableInsertTupleCurrentBatch.
>
OK
>
>> 3) The code in ExecHashJoinOuterGetTuple is hard to understand, it'd
>> very much benefit from some comments. I'm a bit unsure if the likely()
>> and unlikely() hints really help.
>
> In new patch added some comments.
>
> "Likely" and "unlikely" might offer only marginal help on some CPUs and
> might not be beneficial at all on other platforms (I think).
>
Having such hints is an implicit suggestion it's beneficial. Otherwise
we'd just use them everywhere, but we don't - only a tiny fraction of
condition has those.
>
>> 4) Is the hj_outerTupleBuffer buffer really needed / effective? I'd bet
>> just using palloc() will work just as well, thanks to the AllocSet
>> caching etc.
>
> The hj_outerTupleBuffer avoid reform(materialize) tuple in
> non-TTSOpsMinimalTuple scenarios,
>
> see ExecForceStoreMinimalTuple. I added some comments in new patch.
>
AFAIK you mean this comment:
* mtup is hold in hjstate->hj_outerTupleBuffer, so we can using
* shouldFree as false to call ExecForceStoreMinimalTuple().
*
* When slot is TTSOpsMinimalTuple we can avoid realloc memory for
* new MinimalTuple(reuse StringInfo to call ExecHashJoinGetSavedTuple).
But my point was that I don't think the palloc/repalloc should be very
expensive, once the AllocSet warms up a bit.
* More importantly, in non-TTSOpsMinimalTuple scenarios, it can avoid
* reform(materialize) tuple(see ExecForceStoreMinimalTuple).
Yeah, but doesn't that conflate two things - materialization and freeing
the memory? Only because materialization is expensive, is that a good
reason to abandon the memory management too?
>
>> Can you provide more information about the benchmark you did? What
>> hardware, what scale, PostgreSQL configuration, which of the 22 queries
>> are improved, etc.
>>
>> I ran TPC-H with 1GB and 10GB scales on two machines, and I see pretty
>> much no difference compared to master. However, it occurred to me the
>> patch only ever helps if we increase the number of batches during
>> execution, in which case we need to move tuples to the right batch.
>
> Only parallel HashJoin speed up to ~2x(all data cached in memory),
>
> not full query, include non-parallel HashJoin.
>
> non-parallel HashJoin only when batchs large then one will speed up,
>
> because this patch only optimize for read batchs tuples to memory.
>
I'm sorry, but this does not answer *any* of the questions I asked.
Please provide enough info to reproduce the benefit - benchmark scale,
which query, which parameters, etc. Show explain / explain analyze of
the query without / with the patch, stuff like that.
I ran a number of TPC-H benchmarks with the patch and I never a benefit
of this scale.
regards
--
Tomas Vondra
^ permalink raw reply [nested|flat] 80+ messages in thread
* 答复: optimize hashjoin
2024-08-22 12:08 Re: optimize hashjoin bucoo <[email protected]>
2024-08-22 19:26 ` Re: optimize hashjoin Tomas Vondra <[email protected]>
@ 2024-08-23 11:02 ` bucoo <[email protected]>
2024-08-23 12:17 ` Re: optimize hashjoin Robert Haas <[email protected]>
0 siblings, 1 reply; 80+ messages in thread
From: bucoo @ 2024-08-23 11:02 UTC (permalink / raw)
To: 'Tomas Vondra' <[email protected]>; pgsql-hackers; +Cc: 'Tom Lane' <[email protected]>; 'Robert Haas' <[email protected]>
> * mtup is hold in hjstate->hj_outerTupleBuffer, so we can using
> * shouldFree as false to call ExecForceStoreMinimalTuple().
> *
> * When slot is TTSOpsMinimalTuple we can avoid realloc memory for
> * new MinimalTuple(reuse StringInfo to call ExecHashJoinGetSavedTuple).
>
> But my point was that I don't think the palloc/repalloc should be very expensive, once the AllocSet warms up a bit.
Avoiding memory palloc/repalloc is just a side effect of avoiding reform tuple.
> * More importantly, in non-TTSOpsMinimalTuple scenarios, it can avoid
> * reform(materialize) tuple(see ExecForceStoreMinimalTuple).
>
> Yeah, but doesn't that conflate two things - materialization and freeing the memory? Only because materialization is expensive, is that a good reason to abandon the memory management too?
Currently, I haven't thought of a better way to avoid reform.
> >
> >> Can you provide more information about the benchmark you did? What
> >> hardware, what scale, PostgreSQL configuration, which of the 22
> >> queries are improved, etc.
> >>
> >> I ran TPC-H with 1GB and 10GB scales on two machines, and I see
> >> pretty much no difference compared to master. However, it occurred to
> >> me the patch only ever helps if we increase the number of batches
> >> during execution, in which case we need to move tuples to the right batch.
> >
> > Only parallel HashJoin speed up to ~2x(all data cached in memory),
> >
> > not full query, include non-parallel HashJoin.
> >
> > non-parallel HashJoin only when batchs large then one will speed up,
> >
> > because this patch only optimize for read batchs tuples to memory.
> >
>
> I'm sorry, but this does not answer *any* of the questions I asked.
>
> Please provide enough info to reproduce the benefit - benchmark scale, which query, which > parameters, etc. Show explain / explain analyze of the query without / with the patch, stuff > like that.
>
> I ran a number of TPC-H benchmarks with the patch and I never a benefit of this scale.
After further testing, it turns out that the parallel hashjoin did not improve performance. I might have compared it with a debug version at the time. I apologize for that.
Howerver, the non-parallel hashjoin indeed showed about a 10% performance improvement.
Here is the testing information:
CPU: 13th Gen Intel(R) Core(TM) i7-13700
Memory: 32GB
SSD: UMIS REPEYJ512MKN1QWQ
Windows version: win11 23H2 22631.4037
WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2
OS version: rocky linux 9.4
TPCH: SF=8
SQL:
set max_parallel_workers_per_gather = 0;
set enable_mergejoin = off;
explain (verbose,analyze)
select count(*)
from lineitem, orders
where lineitem.l_orderkey = orders.o_orderkey;
patch before:
Aggregate (cost=2422401.83..2422401.84 rows=1 width=8) (actual time=10591.679..10591.681 rows=1 loops=1)
Output: count(*)
-> Hash Join (cost=508496.00..2302429.31 rows=47989008 width=0) (actual time=1075.213..9503.727 rows=47989007 loops=1)
Inner Unique: true
Hash Cond: (lineitem.l_orderkey = orders.o_orderkey)
-> Index Only Scan using lineitem_pkey on public.lineitem (cost=0.56..1246171.69 rows=47989008 width=4) (actual time=0.023..1974.365 rows=47989007 loops=1)
Output: lineitem.l_orderkey
Heap Fetches: 0
-> Hash (cost=311620.43..311620.43 rows=12000000 width=4) (actual time=1074.155..1074.156 rows=12000000 loops=1)
Output: orders.o_orderkey
Buckets: 262144 Batches: 128 Memory Usage: 5335kB
-> Index Only Scan using orders_pkey on public.orders (cost=0.43..311620.43 rows=12000000 width=4) (actual time=0.014..464.346 rows=12000000 loops=1)
Output: orders.o_orderkey
Heap Fetches: 0
Planning Time: 0.141 ms
Execution Time: 10591.730 ms
(16 rows)
Patch after:
Aggregate (cost=2422401.83..2422401.84 rows=1 width=8) (actual time=9826.105..9826.106 rows=1 loops=1)
Output: count(*)
-> Hash Join (cost=508496.00..2302429.31 rows=47989008 width=0) (actual time=1087.588..8726.441 rows=47989007 loops=1)
Inner Unique: true
Hash Cond: (lineitem.l_orderkey = orders.o_orderkey)
-> Index Only Scan using lineitem_pkey on public.lineitem (cost=0.56..1246171.69 rows=47989008 width=4) (actual time=0.015..1989.389 rows=47989007 loops=1)
Output: lineitem.l_orderkey
Heap Fetches: 0
-> Hash (cost=311620.43..311620.43 rows=12000000 width=4) (actual time=1086.357..1086.358 rows=12000000 loops=1)
Output: orders.o_orderkey
Buckets: 262144 Batches: 128 Memory Usage: 5335kB
-> Index Only Scan using orders_pkey on public.orders (cost=0.43..311620.43 rows=12000000 width=4) (actual time=0.011..470.225 rows=12000000 loops=1)
Output: orders.o_orderkey
Heap Fetches: 0
Planning Time: 0.065 ms
Execution Time: 9826.135 ms
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: optimize hashjoin
2024-08-22 12:08 Re: optimize hashjoin bucoo <[email protected]>
2024-08-22 19:26 ` Re: optimize hashjoin Tomas Vondra <[email protected]>
2024-08-23 11:02 ` 答复: optimize hashjoin bucoo <[email protected]>
@ 2024-08-23 12:17 ` Robert Haas <[email protected]>
0 siblings, 0 replies; 80+ messages in thread
From: Robert Haas @ 2024-08-23 12:17 UTC (permalink / raw)
To: bucoo <[email protected]>; +Cc: Tomas Vondra <[email protected]>; pgsql-hackers; Tom Lane <[email protected]>
On Fri, Aug 23, 2024 at 7:02 AM bucoo <[email protected]> wrote:
> Howerver, the non-parallel hashjoin indeed showed about a 10% performance improvement.
> -> Hash Join (cost=508496.00..2302429.31 rows=47989008 width=0) (actual time=1075.213..9503.727 rows=47989007 loops=1)
> -> Hash Join (cost=508496.00..2302429.31 rows=47989008 width=0) (actual time=1087.588..8726.441 rows=47989007 loops=1)
It's not a good idea to test performance with EXPLAIN ANALYZE,
generally speaking. And you usually need to test a few times and
average or something, rather than just a single test. But also, this
doesn't show the hash join being 10% faster. It shows the hash join
being essentially the same speed (1075ms unpatched, 1087ms patched),
and the aggregate node on top of it being faster.
Now, it does seem possible to me that changing one node could cause a
performance improvement for the node above it, but I don't quite see
why that would happen in this case.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 80+ messages in thread
* A typo in .cirrus.tasks.yml
@ 2025-12-31 09:58 Xueyu Gao <[email protected]>
2025-12-31 10:36 ` Re: A typo in .cirrus.tasks.yml Chao Li <[email protected]>
2026-01-07 14:49 ` Re: A typo in .cirrus.tasks.yml Peter Eisentraut <[email protected]>
0 siblings, 2 replies; 80+ messages in thread
From: Xueyu Gao @ 2025-12-31 09:58 UTC (permalink / raw)
To: [email protected]
Hello,
I found a typo in .cirrus.tasks.yml and I have maken a patch.
Please help to confirm.
Thanks,
Xueyu Gao
Attachments:
[application/octet-stream] 0001-PATCH-Fix-the-a-in-.cirrus.tasks.yml.patch (862B, ../../[email protected]/3-0001-PATCH-Fix-the-a-in-.cirrus.tasks.yml.patch)
download | inline diff:
From 9baff3e19c5971bc54f965a4cac2186b6ad1ec1d Mon Sep 17 00:00:00 2001
From: Xueyu Gao <[email protected]>
Date: Wed, 31 Dec 2025 17:27:43 +0800
Subject: [PATCH] Fix the a in .cirrus.tasks.yml
Author: Xueyu Gao <[email protected]>
---
.cirrus.tasks.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml
index 038d043d00e..84ddd014072 100644
--- a/.cirrus.tasks.yml
+++ b/.cirrus.tasks.yml
@@ -636,7 +636,7 @@ task:
env:
CPUS: 4 # always get that much for cirrusci macOS instances
BUILD_JOBS: $CPUS
- # Test performance regresses noticably when using all cores. 8 seems to
+ # Test performance regresses noticeably when using all cores. 8 seems to
# work OK. See
# https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de
TEST_JOBS: 8
--
2.43.0
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: A typo in .cirrus.tasks.yml
2025-12-31 09:58 A typo in .cirrus.tasks.yml Xueyu Gao <[email protected]>
@ 2025-12-31 10:36 ` Chao Li <[email protected]>
1 sibling, 0 replies; 80+ messages in thread
From: Chao Li @ 2025-12-31 10:36 UTC (permalink / raw)
To: Xueyu Gao <[email protected]>; +Cc: [email protected]
> On Dec 31, 2025, at 17:58, Xueyu Gao <[email protected]> wrote:
>
> Hello,
> I found a typo in .cirrus.tasks.yml and I have maken a patch.
“Maken” is also a typo, should be “made”. :)
> Please help to confirm.
Agreed: noticably -> noticeably
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 80+ messages in thread
* Re: A typo in .cirrus.tasks.yml
2025-12-31 09:58 A typo in .cirrus.tasks.yml Xueyu Gao <[email protected]>
@ 2026-01-07 14:49 ` Peter Eisentraut <[email protected]>
1 sibling, 0 replies; 80+ messages in thread
From: Peter Eisentraut @ 2026-01-07 14:49 UTC (permalink / raw)
To: Xueyu Gao <[email protected]>; [email protected]
On 31.12.25 10:58, Xueyu Gao wrote:
> Hello,
> I found a typo in .cirrus.tasks.yml and I have maken a patch.
Committed.
^ permalink raw reply [nested|flat] 80+ messages in thread
end of thread, other threads:[~2026-01-07 14:49 UTC | newest]
Thread overview: 80+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v6 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v5 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2020-12-03 07:54 [PATCH v7 1/2] Add a new OUTDATED filtering facility for REINDEX command. Julien Rouhaud <[email protected]>
2024-08-22 12:08 Re: optimize hashjoin bucoo <[email protected]>
2024-08-22 12:14 ` Re: optimize hashjoin Kirill Reshke <[email protected]>
2024-08-22 19:26 ` Re: optimize hashjoin Tomas Vondra <[email protected]>
2024-08-23 11:02 ` 答复: optimize hashjoin bucoo <[email protected]>
2024-08-23 12:17 ` Re: optimize hashjoin Robert Haas <[email protected]>
2025-12-31 09:58 A typo in .cirrus.tasks.yml Xueyu Gao <[email protected]>
2025-12-31 10:36 ` Re: A typo in .cirrus.tasks.yml Chao Li <[email protected]>
2026-01-07 14:49 ` Re: A typo in .cirrus.tasks.yml Peter Eisentraut <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox