agora inbox for [email protected]help / color / mirror / Atom feed
Re: Changes to error handling for background worker initialization? 20+ messages / 7 participants [nested] [flat]
* Re: Changes to error handling for background worker initialization? @ 2018-11-12 14:32 Jeremy Finzel <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Jeremy Finzel @ 2018-11-12 14:32 UTC (permalink / raw) To: pgsql-hackers On Mon, Oct 22, 2018 at 9:36 AM Jeremy Finzel <[email protected]> wrote: > Hello - > > I have an extension that uses background workers. I pass a database oid > as an argument in order to launch the worker using function > BackgroundWorkerInitializeConnectionByOid. In one of my regression tests > that was written, I intentionally launch the worker with an invalid oid. > In earlier PG versions the worker would successfully launch but then > terminate asynchronously, with a message in the server log. Now, it does > not even successfully launch but immediately errors (hence failing my > regression tests). > > I have recently installed all later point releases of all versions 9.5-11, > so I assume this is due to some code change. The behavior seems reasonable > but I don't find any obvious release notes indicating a patch that would > have changed this behavior. Any thoughts? > > Thanks, > Jeremy > I still haven't determined the source of this error, but I have determined that it must not be related to a difference in point release versions as to background worker error handling, because I am seeing different behavior for identical postgres version on my machine vs. others. I would appreciate any ideas as to how this could possibly happen because I'm not sure the right way now to build this regression test. The test launches the background worker with an invalid database oid. Here is what I am seeing running pg 11.1 on my system (same behavior I get on 9.5-10 as well): SELECT _launch(9999999::OID) AS pid; ! ERROR: could not start background process ! HINT: More details may be available in the server log. This is what others are seeing (the worker fails asynchronously and you see it in the server log): SELECT _launch(9999999::OID) AS pid; ! pid ! ------- ! 18022 ! (1 row) I could share the C code but it's not that interesting. It just calls BackgroundWorkerInitializeConnectionByOid. It is essentially a duplicate of worker_spi. Here is the relevant section: sprintf(worker.bgw_function_name, "worker_spi_main"); snprintf(worker.bgw_name, BGW_MAXLEN, "worker_spi worker %d", i); snprintf(worker.bgw_type, BGW_MAXLEN, "worker_spi"); worker.bgw_main_arg = Int32GetDatum(i); /* set bgw_notify_pid so that we can use WaitForBackgroundWorkerStartup */ worker.bgw_notify_pid = MyProcPid; if (!RegisterDynamicBackgroundWorker(&worker, &handle)) PG_RETURN_NULL(); status = WaitForBackgroundWorkerStartup(handle, &pid); if (status == BGWH_STOPPED) ereport(ERROR, (errcode(ERRCODE_INSUFFICIENT_RESOURCES), errmsg("could not start background process"), errhint("More details may be available in the server log."))); So on my machine, I am getting status == BGWH_STOPPED, whereas with others, they are not getting that behavior. Thanks, Jeremy ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v18 3/6] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 188 +++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 203 ++++++++++++-- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 256 +++++++++++++++++- src/backend/catalog/pg_type.c | 69 +++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 32 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 +++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 25 +- src/include/catalog/index.h | 4 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 ++++++ .../regress/expected/collate.icu.utf8.out | 157 +++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 106 ++++++++ 25 files changed, 1156 insertions(+), 88 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index f26e245c65..98313e2b05 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21122,7 +21122,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 281f5b7c28..ddcfb7a856 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,81 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, + *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1670,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1605,8 +1688,8 @@ recordDependencyOnExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1633,12 +1716,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1694,8 +1783,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, self_addrs->refs, self_addrs->numrefs, - NULL, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1716,8 +1805,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1739,8 +1828,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1773,6 +1867,49 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* + * Record collations from the type itself, or underlying in case + * of complex type. Note that if the direct parent is a + * CollateExpr node, there's no need to record the type underlying + * collation if any. A dependency already exists for the owning + * relation, and a change in the collation sort order wouldn't + * cause any harm as the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + + /* + * otherwise, it may be a composite type having underlying + * collations + */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1798,10 +1935,12 @@ find_expr_references_walker(Node *node, * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * case where the collation is "default", since we know that's pinned, + * if the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1891,7 +2030,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1979,7 +2119,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2010,7 +2151,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2023,7 +2165,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2036,7 +2179,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2125,7 +2269,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2270,7 +2415,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2292,7 +2439,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2688,8 +2837,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, referenced->numrefs, NULL, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..b756af35d0 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */ , + false /* don't track versions */ ); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 2d81bc3cbc..a169b707a0 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,77 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or + * not, removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1206,30 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicated entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicated entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1301,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -1676,14 +1836,9 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName) /* * Swap all dependencies of and on the old index to the new one, and - * vice-versa. Note that a call to CommandCounterIncrement() would cause - * duplicate entries in pg_depend, so this should not be done. + * vice-versa. */ - changeDependenciesOf(RelationRelationId, newIndexId, oldIndexId); - changeDependenciesOn(RelationRelationId, newIndexId, oldIndexId); - - changeDependenciesOf(RelationRelationId, oldIndexId, newIndexId); - changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId); + swapDependencies(RelationRelationId, newIndexId, oldIndexId); /* * Copy over statistics from old to new index @@ -2635,6 +2790,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3789,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 84a0b99311..7f5375fd14 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,6 +19,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" @@ -27,9 +28,18 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static long changeDependenciesOf(Oid classId, Oid oldObjectId, + Oid newObjectId, bool preserve_version); +static long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, + Oid newRefObjectId); +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); +static char *getDependencyVersion(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -45,19 +55,48 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, 1, NULL, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); +} + +/* + * Given a list of collations, record a dependency on its underlying collation + * version. + */ +void +recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) +{ + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -65,6 +104,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -83,12 +123,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries + * and calling CommandCounterIncrement() if the dependencies + * are registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -402,16 +479,33 @@ changeDependencyFor(Oid classId, Oid objectId, } /* - * Adjust all dependency records to come from a different object of the same type + * Swap all dependencies of and on the old index to the new one, and + * vice-versa, while preserving any referenced version for the original owners. + * Note that a call to CommandCounterIncrement() would cause duplicate entries + * in pg_depend, so this should not be done. + */ +void +swapDependencies(Oid classId, Oid firstObjectId, Oid secondObjectId) +{ + changeDependenciesOf(classId, firstObjectId, secondObjectId, true); + changeDependenciesOn(classId, firstObjectId, secondObjectId); + + changeDependenciesOf(classId, secondObjectId, firstObjectId, true); + changeDependenciesOn(classId, secondObjectId, firstObjectId); +} + +/* + * Adjust all dependency records to come from a different object of the same + * type, optionally preserving the original referenced version. * * classId/oldObjectId specify the old referencing object. * newObjectId is the new referencing object (must be of class classId). * * Returns the number of records updated. */ -long +static long changeDependenciesOf(Oid classId, Oid oldObjectId, - Oid newObjectId) + Oid newObjectId, bool preserve_version) { long count = 0; Relation depRel; @@ -436,13 +530,49 @@ changeDependenciesOf(Oid classId, Oid oldObjectId, while (HeapTupleIsValid((tup = systable_getnext(scan)))) { Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup); + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + bool isnull = true; - /* make a modifiable copy */ - tup = heap_copytuple(tup); - depform = (Form_pg_depend) GETSTRUCT(tup); + memset(values, 0, sizeof(values)); + memset(nulls, 0, sizeof(nulls)); + memset(replaces, 0, sizeof(replaces)); - depform->objid = newObjectId; + values[Anum_pg_depend_objid - 1] = newObjectId; + replaces[Anum_pg_depend_objid - 1] = true; + /* + * We assume that a version would exist for both the old and new + * object or none. + */ + if (preserve_version) + { + heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + } + + if (!isnull) + { + ObjectAddress depender, + referenced; + char *version; + + ObjectAddressSubSet(depender, depform->classid, + newObjectId, depform->objsubid); + ObjectAddressSubSet(referenced, depform->refclassid, + depform->refobjid, depform->refobjsubid); + + version = getDependencyVersion(&depender, &referenced); + if (version) + values[Anum_pg_depend_refobjversion - 1] = CStringGetTextDatum(version); + else + nulls[Anum_pg_depend_refobjversion - 1] = true; + replaces[Anum_pg_depend_refobjversion - 1] = true; + } + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); CatalogTupleUpdate(depRel, &tup->t_self, tup); heap_freetuple(tup); @@ -465,7 +595,7 @@ changeDependenciesOf(Oid classId, Oid oldObjectId, * * Returns the number of records updated. */ -long +static long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, Oid newRefObjectId) { @@ -545,6 +675,104 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenced addresses. + */ +static bool +dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + +static char * +getDependencyVersion(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + Datum depversion; + char *version = NULL; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + bool isnull; + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + version = isnull ? NULL : TextDatumGetCString(depversion); + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return version; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..2b62e6f47c 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..8f92601865 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,36 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL, as VACUUM FULL will recreate the index and + * update the recorded collation version. + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..28ef63d6ab 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..fc3a41221f 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 62e86a19b6..4d99f0e6da 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,17 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); + extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); @@ -201,11 +214,7 @@ extern long changeDependencyFor(Oid classId, Oid objectId, Oid refClassId, Oid oldRefObjectId, Oid newRefObjectId); -extern long changeDependenciesOf(Oid classId, Oid oldObjectId, - Oid newObjectId); - -extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, - Oid newRefObjectId); +extern void swapDependencies(Oid classId, Oid firstObjectId, Oid secondObjectId); extern Oid getExtensionOfObject(Oid classId, Oid objectId); extern List *getAutoExtensionsOfObject(Oid classId, Oid objectId); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..0899a203c0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, @@ -131,6 +133,8 @@ extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); extern void index_set_state_flags(Oid indexId, IndexStateFlagsAction action); +void index_update_collation_versions(Oid relid); + extern void reindex_index(Oid indexId, bool skip_constraint_checks, char relpersistence, int options); diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 7b37562648..2d511c5cba 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -361,6 +361,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..82005e9eba 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..db386c1b09 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,163 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..e93530af55 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,112 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --vtzGhvizbBRQ85DL Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v18-0004-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v17 4/7] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 188 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 194 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 132 +++++++++++- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/indexcmds.c | 11 + src/backend/commands/vacuum.c | 32 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 19 +- src/include/catalog/index.h | 4 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 157 ++++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 106 ++++++++++ 26 files changed, 1048 insertions(+), 68 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index b84502295e..f919561923 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21123,7 +21123,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 281f5b7c28..ddcfb7a856 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,81 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, + *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1670,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1605,8 +1688,8 @@ recordDependencyOnExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1633,12 +1716,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1694,8 +1783,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, self_addrs->refs, self_addrs->numrefs, - NULL, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1716,8 +1805,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1739,8 +1828,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1773,6 +1867,49 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* + * Record collations from the type itself, or underlying in case + * of complex type. Note that if the direct parent is a + * CollateExpr node, there's no need to record the type underlying + * collation if any. A dependency already exists for the owning + * relation, and a change in the collation sort order wouldn't + * cause any harm as the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + + /* + * otherwise, it may be a composite type having underlying + * collations + */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1798,10 +1935,12 @@ find_expr_references_walker(Node *node, * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * case where the collation is "default", since we know that's pinned, + * if the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1891,7 +2030,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1979,7 +2119,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2010,7 +2151,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2023,7 +2165,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2036,7 +2179,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2125,7 +2269,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2270,7 +2415,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2292,7 +2439,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2688,8 +2837,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, referenced->numrefs, NULL, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..b756af35d0 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */ , + false /* don't track versions */ ); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 76fd938ce3..11a998d3d4 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,77 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or + * not, removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1206,30 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicated entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicated entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1301,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2635,6 +2795,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3794,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 84a0b99311..a17bbe4442 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,6 +19,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" @@ -27,9 +28,12 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -45,19 +49,48 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, 1, NULL, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); +} + +/* + * Given a list of collations, record a dependency on its underlying collation + * version. + */ +void +recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) +{ + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -65,6 +98,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -83,12 +117,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries + * and calling CommandCounterIncrement() if the dependencies + * are registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -545,6 +616,55 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenced addresses. + */ +static bool +dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..2b62e6f47c 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4e8263af4b..8f42da2444 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -3323,6 +3323,17 @@ ReindexRelationConcurrently(Oid relationOid, int options) /* Start a new transaction to finish process properly */ StartTransactionCommand(); + /* + * Record the current versions of all depended-on collations, for all new + * indexes. + */ + foreach(lc, newIndexIds) + { + Oid indOid = lfirst_oid(lc); + + index_update_collation_versions(indOid); + } + /* Log what we did */ if (options & REINDEXOPT_VERBOSE) { diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..8f92601865 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,36 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL, as VACUUM FULL will recreate the index and + * update the recorded collation version. + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..28ef63d6ab 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..fc3a41221f 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 62e86a19b6..e39e362524 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,17 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); + extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..0899a203c0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, @@ -131,6 +133,8 @@ extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); extern void index_set_state_flags(Oid indexId, IndexStateFlagsAction action); +void index_update_collation_versions(Oid relid); + extern void reindex_index(Oid indexId, bool skip_constraint_checks, char relpersistence, int options); diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 97890946c5..3eb5ec4fcd 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -350,6 +350,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..82005e9eba 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..db386c1b09 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,163 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..e93530af55 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,112 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --FCuugMFkClbJLl1L Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v17-0005-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH 4/6] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. That version is checked against current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already beed done to avoid checking and reporting the message multiple time in a backend lifetime. Author: Thomas Munro, Julien Rouhaud Reviewed-by: Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- src/backend/catalog/dependency.c | 189 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 191 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 136 +++++++++++-- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 31 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 22 +- src/include/catalog/index.h | 2 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 130 ++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 83 ++++++++ 24 files changed, 968 insertions(+), 83 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index b2d991ac7f..8414f2bfd9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21115,7 +21115,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 78c31baa34..93f57cd633 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -77,6 +77,7 @@ #include "parser/parsetree.h" #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -137,6 +138,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -437,6 +441,80 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1590,6 +1668,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1602,9 +1684,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* And record 'em */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1631,12 +1714,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1690,9 +1779,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) recordMultipleDependencies(depender, - self_addrs->refs, NULL, + self_addrs->refs, self_addrs->numrefs, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1711,9 +1801,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the external dependencies */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1735,8 +1826,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1769,6 +1865,46 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* Record collations from the type itself, or underlying in case of + * complex type. Note that if the direct parent is a CollateExpr + * node, there's no need to record the type underlying collation if + * any. A dependency already exists for the owning relation, and a + * change in the collation sort order wouldn't cause any harm as + * the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + /* + * otherwise, it may be a composite type having underlying + * collations */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1793,11 +1929,13 @@ find_expr_references_walker(Node *node, /* * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to - * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * a simple constant. However we can save work in the most common case + * where the collation is "default", since we know that's pinned, if + * the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1887,7 +2025,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1975,7 +2114,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2006,7 +2146,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2019,7 +2160,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2032,7 +2174,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2121,7 +2264,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2266,7 +2410,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2288,7 +2434,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2684,8 +2832,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, NULL, referenced->numrefs, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index e31478bf91..46e9d74a97 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */, + false /* don't track versions */); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1681f61727..9f9281a7af 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,75 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or not, + * removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1204,29 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1298,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +static void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2634,6 +2791,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3611,6 +3779,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 3d2b1cc911..0eaefbd032 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 7fdbdf0ae8..7e087fefc4 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,16 +19,21 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" #include "commands/extension.h" #include "miscadmin.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -44,34 +49,47 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, NULL, 1, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); } /* - * As recordDependencyOn(), but also capture a version string so that changes - * in the referenced object can be detected. The meaning of the version - * string depends on the referenced object. Currently it is used for - * detecting changes in collation versions. + * Given a list of collations, record a dependency on its underlying collation + * version. */ -void -recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior) +void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) { - recordMultipleDependencies(depender, referenced, version, 1, behavior); + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -79,6 +97,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -97,12 +116,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries and + * calling CommandCounterIncrement() if the dependencies are + * registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -559,6 +615,54 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenceds addresses. + */ +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index 8d7572da51..6e91518694 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -511,6 +512,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 493aa21a14..05709fab77 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -272,28 +272,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..2b7de111cd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,35 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..f2fc427fc8 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..18a2d72f91 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 77cf0612ed..8750bfc36f 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,22 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); -extern void recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..c619d02465 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index e1a5ab3df3..2bf6f868a5 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -335,6 +335,8 @@ extern void GenerateTypeDependencies(Oid typeObjectId, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..3656ea94e8 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..09512c0f66 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,136 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..c8f1a620d2 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,89 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --mP3DRpeJDSE+ciuQ Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename*0=0005-Preserve-index-dependencies-on-collation-during-pg_u-v12; filename*1=".patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v16 4/7] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 189 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 191 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 136 +++++++++++-- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 31 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 22 +- src/include/catalog/index.h | 2 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 130 ++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 83 ++++++++ 25 files changed, 974 insertions(+), 83 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 02779ab713..d28995dd5a 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21125,7 +21125,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 660033b9c1..c0e36ef4cd 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,80 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1669,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1603,9 +1685,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* And record 'em */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1632,12 +1715,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1691,9 +1780,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) recordMultipleDependencies(depender, - self_addrs->refs, NULL, + self_addrs->refs, self_addrs->numrefs, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1712,9 +1802,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the external dependencies */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1736,8 +1827,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1770,6 +1866,46 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* Record collations from the type itself, or underlying in case of + * complex type. Note that if the direct parent is a CollateExpr + * node, there's no need to record the type underlying collation if + * any. A dependency already exists for the owning relation, and a + * change in the collation sort order wouldn't cause any harm as + * the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + /* + * otherwise, it may be a composite type having underlying + * collations */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1794,11 +1930,13 @@ find_expr_references_walker(Node *node, /* * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to - * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * a simple constant. However we can save work in the most common case + * where the collation is "default", since we know that's pinned, if + * the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1888,7 +2026,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1976,7 +2115,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2007,7 +2147,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2020,7 +2161,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2033,7 +2175,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2122,7 +2265,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2267,7 +2411,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2289,7 +2435,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2685,8 +2833,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, NULL, referenced->numrefs, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..4546789446 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */, + false /* don't track versions */); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 76fd938ce3..369b5d43f1 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,75 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or not, + * removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1204,29 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1298,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +static void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2635,6 +2792,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3791,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index b7beb2884e..3f113499b2 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,16 +19,21 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" #include "commands/extension.h" #include "miscadmin.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -44,34 +49,47 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, NULL, 1, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); } /* - * As recordDependencyOn(), but also capture a version string so that changes - * in the referenced object can be detected. The meaning of the version - * string depends on the referenced object. Currently it is used for - * detecting changes in collation versions. + * Given a list of collations, record a dependency on its underlying collation + * version. */ -void -recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior) +void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) { - recordMultipleDependencies(depender, referenced, version, 1, behavior); + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -79,6 +97,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -97,12 +116,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries and + * calling CommandCounterIncrement() if the dependencies are + * registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -559,6 +615,54 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenceds addresses. + */ +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..fe14b6bd38 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..2b7de111cd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,35 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..f2fc427fc8 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..18a2d72f91 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 7d9fc866b9..8858ceb3df 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,22 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); -extern void recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..c619d02465 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 97890946c5..3eb5ec4fcd 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -350,6 +350,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..3656ea94e8 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..09512c0f66 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,136 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..c8f1a620d2 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,89 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --sdtB3X0nJg68CQEu Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v16-0005-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v15 4/7] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 189 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 191 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 136 +++++++++++-- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 31 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 22 +- src/include/catalog/index.h | 2 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 130 ++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 83 ++++++++ 25 files changed, 974 insertions(+), 83 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index b2d991ac7f..8414f2bfd9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21115,7 +21115,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 660033b9c1..c0e36ef4cd 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,80 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1669,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1603,9 +1685,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* And record 'em */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1632,12 +1715,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1691,9 +1780,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) recordMultipleDependencies(depender, - self_addrs->refs, NULL, + self_addrs->refs, self_addrs->numrefs, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1712,9 +1802,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the external dependencies */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1736,8 +1827,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1770,6 +1866,46 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* Record collations from the type itself, or underlying in case of + * complex type. Note that if the direct parent is a CollateExpr + * node, there's no need to record the type underlying collation if + * any. A dependency already exists for the owning relation, and a + * change in the collation sort order wouldn't cause any harm as + * the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + /* + * otherwise, it may be a composite type having underlying + * collations */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1794,11 +1930,13 @@ find_expr_references_walker(Node *node, /* * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to - * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * a simple constant. However we can save work in the most common case + * where the collation is "default", since we know that's pinned, if + * the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1888,7 +2026,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1976,7 +2115,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2007,7 +2147,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2020,7 +2161,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2033,7 +2175,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2122,7 +2265,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2267,7 +2411,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2289,7 +2435,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2685,8 +2833,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, NULL, referenced->numrefs, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..4546789446 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */, + false /* don't track versions */); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 76fd938ce3..369b5d43f1 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,75 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or not, + * removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1204,29 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1298,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +static void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2635,6 +2792,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3791,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index b7beb2884e..3f113499b2 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,16 +19,21 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" #include "commands/extension.h" #include "miscadmin.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -44,34 +49,47 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, NULL, 1, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); } /* - * As recordDependencyOn(), but also capture a version string so that changes - * in the referenced object can be detected. The meaning of the version - * string depends on the referenced object. Currently it is used for - * detecting changes in collation versions. + * Given a list of collations, record a dependency on its underlying collation + * version. */ -void -recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior) +void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) { - recordMultipleDependencies(depender, referenced, version, 1, behavior); + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -79,6 +97,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -97,12 +116,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries and + * calling CommandCounterIncrement() if the dependencies are + * registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -559,6 +615,54 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenceds addresses. + */ +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..fe14b6bd38 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..2b7de111cd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,35 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..f2fc427fc8 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..18a2d72f91 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 7d9fc866b9..8858ceb3df 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,22 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); -extern void recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..c619d02465 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 97890946c5..3eb5ec4fcd 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -350,6 +350,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..3656ea94e8 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..09512c0f66 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,136 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..c8f1a620d2 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,89 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --bp/iNruPH9dso1Pn Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v15-0005-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v14 4/7] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 189 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 191 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 136 +++++++++++-- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 31 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 22 +- src/include/catalog/index.h | 2 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 130 ++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 83 ++++++++ 25 files changed, 974 insertions(+), 83 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index b2d991ac7f..8414f2bfd9 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21115,7 +21115,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 660033b9c1..c0e36ef4cd 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,80 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1669,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1603,9 +1685,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* And record 'em */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1632,12 +1715,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1691,9 +1780,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) recordMultipleDependencies(depender, - self_addrs->refs, NULL, + self_addrs->refs, self_addrs->numrefs, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1712,9 +1802,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the external dependencies */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1736,8 +1827,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1770,6 +1866,46 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* Record collations from the type itself, or underlying in case of + * complex type. Note that if the direct parent is a CollateExpr + * node, there's no need to record the type underlying collation if + * any. A dependency already exists for the owning relation, and a + * change in the collation sort order wouldn't cause any harm as + * the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + /* + * otherwise, it may be a composite type having underlying + * collations */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1794,11 +1930,13 @@ find_expr_references_walker(Node *node, /* * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to - * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * a simple constant. However we can save work in the most common case + * where the collation is "default", since we know that's pinned, if + * the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1888,7 +2026,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1976,7 +2115,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2007,7 +2147,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2020,7 +2161,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2033,7 +2175,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2122,7 +2265,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2267,7 +2411,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2289,7 +2435,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2685,8 +2833,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, NULL, referenced->numrefs, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..4546789446 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */, + false /* don't track versions */); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 76fd938ce3..369b5d43f1 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,75 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or not, + * removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1204,29 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1298,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +static void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2635,6 +2792,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3791,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 7fdbdf0ae8..7e087fefc4 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,16 +19,21 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" #include "commands/extension.h" #include "miscadmin.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -44,34 +49,47 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, NULL, 1, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); } /* - * As recordDependencyOn(), but also capture a version string so that changes - * in the referenced object can be detected. The meaning of the version - * string depends on the referenced object. Currently it is used for - * detecting changes in collation versions. + * Given a list of collations, record a dependency on its underlying collation + * version. */ -void -recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior) +void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) { - recordMultipleDependencies(depender, referenced, version, 1, behavior); + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -79,6 +97,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -97,12 +116,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries and + * calling CommandCounterIncrement() if the dependencies are + * registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -559,6 +615,54 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenceds addresses. + */ +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..fe14b6bd38 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..2b7de111cd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,35 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..f2fc427fc8 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..18a2d72f91 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 77cf0612ed..8750bfc36f 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,22 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); -extern void recordDependencyOnVersion(const ObjectAddress *depender, - const ObjectAddress *referenced, - const NameData *version, - DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const NameData *version, int nreferenced, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..c619d02465 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 97890946c5..3eb5ec4fcd 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -350,6 +350,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..3656ea94e8 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..09512c0f66 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,136 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..c8f1a620d2 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,89 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v14-0005-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v16 4/7] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. A new flag is added in RelationData to specify that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro and Julien Rouhaud Reviewed-by: Peter Eisentraut, Laurenz Albe Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 189 +++++++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 191 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 130 +++++++++++- src/backend/catalog/pg_type.c | 69 +++++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 31 +++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++++- src/backend/utils/cache/relcache.c | 2 + src/include/catalog/dependency.h | 19 +- src/include/catalog/index.h | 2 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++++ .../regress/expected/collate.icu.utf8.out | 130 ++++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 83 ++++++++ 25 files changed, 979 insertions(+), 69 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index fd1bba9959..29e4a9b73e 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21125,7 +21125,7 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); <para> <function>pg_collation_actual_version</function> returns the actual version of the collation object as it is currently installed in the - operating system. + operating system. An empty string is returned if the version is unknown. </para> <para> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index c54a7c420d..7209a606fa 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 660033b9c1..c0e36ef4cd 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -78,6 +78,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -138,6 +139,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -438,6 +442,80 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1591,6 +1669,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1603,9 +1685,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, /* And record 'em */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1632,12 +1715,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1691,9 +1780,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the self-dependencies with the appropriate direction */ if (!reverse_self) recordMultipleDependencies(depender, - self_addrs->refs, NULL, + self_addrs->refs, self_addrs->numrefs, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1712,9 +1802,10 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, /* Record the external dependencies */ recordMultipleDependencies(depender, - context.addrs->refs, NULL, + context.addrs->refs, context.addrs->numrefs, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1736,8 +1827,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1770,6 +1866,46 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* Record collations from the type itself, or underlying in case of + * complex type. Note that if the direct parent is a CollateExpr + * node, there's no need to record the type underlying collation if + * any. A dependency already exists for the owning relation, and a + * change in the collation sort order wouldn't cause any harm as + * the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + /* + * otherwise, it may be a composite type having underlying + * collations */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1794,11 +1930,13 @@ find_expr_references_walker(Node *node, /* * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to - * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * a simple constant. However we can save work in the most common case + * where the collation is "default", since we know that's pinned, if + * the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1888,7 +2026,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1976,7 +2115,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2007,7 +2147,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2020,7 +2161,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2033,7 +2175,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2122,7 +2265,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2267,7 +2411,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2289,7 +2435,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2685,8 +2833,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, NULL, referenced->numrefs, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 9d9e915979..4546789446 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2304,7 +2304,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2314,7 +2314,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3638,7 +3638,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */, + false /* don't track versions */); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 76fd938ce3..369b5d43f1 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -52,6 +52,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -74,6 +75,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/snapmgr.h" @@ -117,6 +119,7 @@ static void UpdateIndexRelation(Oid indexoid, Oid heapoid, bool immediate, bool isvalid, bool isready); +static bool index_depends_stable_coll_order(Oid amoid); static void index_update_stats(Relation rel, bool hasindex, double reltuples); @@ -1025,6 +1028,10 @@ index_create(Relation heapRelation, { ObjectAddress myself, referenced; + ListCell *lc; + List *colls = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; myself.classId = RelationRelationId; myself.objectId = indexRelationId; @@ -1115,21 +1122,75 @@ index_create(Relation heapRelation, recordDependencyOn(&myself, &referenced, DEPENDENCY_PARTITION_SEC); } - /* Store dependency on collations */ - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) + colls = lappend_oid(colls, colloid); + else { - referenced.classId = CollationRelationId; - referenced.objectId = collationObjectId[i]; - referenced.objectSubId = 0; + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); - recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL); + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* + * Then split the dependencies on whether they're deterministic or not, + * removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = list_append_unique_oid(determ_colls, c); + else + nondeterm_colls = list_append_unique_oid(nondeterm_colls, c); + } + + /* + * For deterministic transaction, only track the version if the AM + * relies on a stable ordering. + */ + if (determ_colls) + { + bool track_version; + + track_version = index_depends_stable_coll_order(indexInfo->ii_Am); + + recordDependencyOnCollations(&myself, determ_colls, track_version); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1143,21 +1204,29 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicate entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1229,6 +1298,94 @@ index_create(Relation heapRelation, return indexRelationId; } +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +static void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -2635,6 +2792,17 @@ FormIndexDatum(IndexInfo *indexInfo, elog(ERROR, "wrong number of index expressions"); } +/* + * Returns whether the given index access method depend on a stable collation + * order. + */ +static bool +index_depends_stable_coll_order(Oid amoid) +{ + return (amoid != HASH_AM_OID && + strcmp(get_am_name(amoid), "bloom") != 0); +} + /* * index_update_stats --- update pg_class entry after CREATE INDEX or REINDEX @@ -3623,6 +3791,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 90932be831..3cc0f6651c 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -360,7 +360,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 312d4d3d90..3f113499b2 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,6 +19,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" @@ -27,9 +28,12 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -45,19 +49,47 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, NULL, 1, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); +} + +/* + * Given a list of collations, record a dependency on its underlying collation + * version. + */ +void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) +{ + ListCell *lc; + + foreach(lc, collations) + { + ObjectAddress referenced; + + referenced.classId = CollationRelationId; + referenced.objectId = lfirst_oid(lc); + referenced.objectSubId = 0; + + recordMultipleDependencies(myself, &referenced, 1, + DEPENDENCY_NORMAL, record_version); + } } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const char *version, int nreferenced, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -65,6 +97,7 @@ recordMultipleDependencies(const ObjectAddress *depender, int i; bool nulls[Natts_pg_depend]; Datum values[Natts_pg_depend]; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -83,12 +116,49 @@ recordMultipleDependencies(const ObjectAddress *depender, for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant depedencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries and + * calling CommandCounterIncrement() if the dependencies are + * registered in multiple calls. + */ + if (dependencyExists(depender, referenced)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + version = get_collation_version_for_oid(referenced->objectId); + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (!isObjectPinned(referenced, dependDesc)) + if (ignore_systempin || !isObjectPinned(referenced, dependDesc)) { /* * Record the Dependency. Note we don't bother to check for @@ -545,6 +615,54 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenceds addresses. + */ +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + ret = true; + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index cd56714968..fe14b6bd38 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 78eceda848..d4eccf163f 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -273,28 +273,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index d625d17bf4..2b7de111cd 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -634,6 +636,35 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index d82fc5ab8b..f2fc427fc8 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 60dab33fcb..18a2d72f91 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -148,6 +150,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1473,7 +1478,7 @@ pg_newlocale_from_collation(Oid collid) * NULL (if it doesn't support versions). It must not return NULL for some * collcollate and not NULL for others. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1511,6 +1516,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index ff70326474..bdf50ffe89 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -41,6 +41,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/pg_am.h" @@ -5623,6 +5624,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 473342dad2..8858ceb3df 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -156,7 +156,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -176,17 +177,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); + extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, - const char *version, int nreferenced, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a2890c1314..c619d02465 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,8 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 97890946c5..3eb5ec4fcd 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -350,6 +350,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 44ed04dd3f..3656ea94e8 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /* * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..09512c0f66 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,136 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +---------------------------+------------+--------------------- + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(49 rows) + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index ae95bb38a6..94b4daf4d6 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,15 +2065,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2093,15 +2095,17 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(9 rows) +(11 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..c8f1a620d2 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,89 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' collate custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' collate custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' collate "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --fUYQa+Pmc3FrFX/N Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v16-0005-Preserve-index-dependencies-on-collation-during-.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH 1/4] drop has oids field instead of having static values @ 2023-07-26 19:31 Pierre Ducroquet <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Pierre Ducroquet @ 2023-07-26 19:31 UTC (permalink / raw) --- src/bin/pg_dump/pg_backup_archiver.c | 3 +-- src/bin/pg_dump/pg_backup_archiver.h | 3 ++- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 4d83381d84..f4c782d63d 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -2510,7 +2510,6 @@ WriteToc(ArchiveHandle *AH) WriteStr(AH, te->tablespace); WriteStr(AH, te->tableam); WriteStr(AH, te->owner); - WriteStr(AH, "false"); /* Dump list of dependencies */ for (i = 0; i < te->nDeps; i++) @@ -2618,7 +2617,7 @@ ReadToc(ArchiveHandle *AH) is_supported = true; if (AH->version < K_VERS_1_9) is_supported = false; - else + else if (AH->version < K_VERS_1_16) { tmp = ReadStr(AH); diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h index b07673933d..b78e326f73 100644 --- a/src/bin/pg_dump/pg_backup_archiver.h +++ b/src/bin/pg_dump/pg_backup_archiver.h @@ -68,10 +68,11 @@ #define K_VERS_1_15 MAKE_ARCHIVE_VERSION(1, 15, 0) /* add * compression_algorithm * in header */ +#define K_VERS_1_16 MAKE_ARCHIVE_VERSION(1, 16, 0) /* optimize dump format */ /* Current archive version number (the format we can output) */ #define K_VERS_MAJOR 1 -#define K_VERS_MINOR 15 +#define K_VERS_MINOR 16 #define K_VERS_REV 0 #define K_VERS_SELF MAKE_ARCHIVE_VERSION(K_VERS_MAJOR, K_VERS_MINOR, K_VERS_REV) -- 2.42.0 --nextPart4294444.ejJDZkT8p0 Content-Disposition: attachment; filename="0002-convert-sscanf-to-strtoul.patch" Content-Transfer-Encoding: 7Bit Content-Type: text/x-patch; charset="x-UTF_8J"; name="0002-convert-sscanf-to-strtoul.patch" ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants @ 2024-05-24 20:00 Tom Lane <[email protected]> 2024-05-25 14:09 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 0 siblings, 2 replies; 20+ messages in thread From: Tom Lane @ 2024-05-24 20:00 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Robert Haas <[email protected]> writes: > On Fri, May 24, 2024 at 2:57 PM Tom Lane <[email protected]> wrote: >> Doesn't seem right to me. That will give pg_dump the wrong idea >> of what the initial privileges actually were, and I don't see how >> it can construct correct delta GRANT/REVOKE on the basis of false >> information. During the dump reload, the extension will be >> recreated with the original owner (I think), causing its objects' >> privileges to go back to the original pg_init_privs values. > Oh! That does seem like it would make what I said wrong, but how would > it even know who the original owner was? Shouldn't we be recreating > the object with the owner it had at dump time? Keep in mind that the whole point here is for the pg_dump script to just say "CREATE EXTENSION foo", not to mess with the individual objects therein. So the objects are (probably) going to be owned by the user that issued CREATE EXTENSION. In the original conception, that was the end of it: what you got for the member objects was whatever state CREATE EXTENSION left behind. The idea of pg_init_privs is to support dump/reload of subsequent manual alterations of privileges for extension-created objects. I'm not, at this point, 100% certain that that's a fully realizable goal. But I definitely think it's insane to expect that to work without also tracking changes in the ownership of said objects. Maybe forbidding ALTER OWNER on extension-owned objects isn't such a bad idea? regards, tom lane ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> @ 2024-05-25 14:09 ` Hannu Krosing <[email protected]> 2024-05-25 14:47 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 1 sibling, 1 reply; 20+ messages in thread From: Hannu Krosing @ 2024-05-25 14:09 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Daniel Gustafsson <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Fri, May 24, 2024 at 10:00 PM Tom Lane <[email protected]> wrote: > > Robert Haas <[email protected]> writes: > > On Fri, May 24, 2024 at 2:57 PM Tom Lane <[email protected]> wrote: > >> Doesn't seem right to me. That will give pg_dump the wrong idea > >> of what the initial privileges actually were, and I don't see how > >> it can construct correct delta GRANT/REVOKE on the basis of false > >> information. During the dump reload, the extension will be > >> recreated with the original owner (I think), causing its objects' > >> privileges to go back to the original pg_init_privs values. > > > Oh! That does seem like it would make what I said wrong, but how would > > it even know who the original owner was? Shouldn't we be recreating > > the object with the owner it had at dump time? > > Keep in mind that the whole point here is for the pg_dump script to > just say "CREATE EXTENSION foo", not to mess with the individual > objects therein. So the objects are (probably) going to be owned by > the user that issued CREATE EXTENSION. > > In the original conception, that was the end of it: what you got for > the member objects was whatever state CREATE EXTENSION left behind. > The idea of pg_init_privs is to support dump/reload of subsequent > manual alterations of privileges for extension-created objects. > I'm not, at this point, 100% certain that that's a fully realizable > goal. The issue became visible because pg_dump issued a bogus REVOKE ALL ON TABLE public.pg_stat_statements FROM "16390"; Maybe the right place for a fix is in pg_dump and the fix would be to *not* issue REVOKE ALL ON <any object> FROM <non-existing users> ? Or alternatively change REVOKE to treat non-existing users as a no-op ? Also, the pg_init_privs entry should either go away or at least be changed at the point when the user referenced in init-privs is dropped. Having an pg_init_privs entry referencing a non-existing user is certainly of no practical use. Or maybe we should change the user at that point to NULL or some special non-existing-user-id ? > But I definitely think it's insane to expect that to work > without also tracking changes in the ownership of said objects. > > Maybe forbidding ALTER OWNER on extension-owned objects isn't > such a bad idea? ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-05-25 14:09 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> @ 2024-05-25 14:47 ` Tom Lane <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Tom Lane @ 2024-05-25 14:47 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Robert Haas <[email protected]>; Daniel Gustafsson <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Hannu Krosing <[email protected]> writes: > Having an pg_init_privs entry referencing a non-existing user is > certainly of no practical use. Sure, that's not up for debate. What I think we're discussing right now is 1. What other cases are badly handled by the pg_init_privs mechanisms. 2. How much of that is practical to fix in v17, seeing that it's all long-standing bugs and we're already past beta1. I kind of doubt that the answer to #2 is "all of it". But perhaps we can do better than "none of it". regards, tom lane ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> @ 2024-06-04 14:16 ` Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 1 sibling, 1 reply; 20+ messages in thread From: Robert Haas @ 2024-06-04 14:16 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> On Fri, May 24, 2024 at 4:00 PM Tom Lane <[email protected]> wrote: > > Oh! That does seem like it would make what I said wrong, but how would > > it even know who the original owner was? Shouldn't we be recreating > > the object with the owner it had at dump time? > > Keep in mind that the whole point here is for the pg_dump script to > just say "CREATE EXTENSION foo", not to mess with the individual > objects therein. So the objects are (probably) going to be owned by > the user that issued CREATE EXTENSION. > > In the original conception, that was the end of it: what you got for > the member objects was whatever state CREATE EXTENSION left behind. > The idea of pg_init_privs is to support dump/reload of subsequent > manual alterations of privileges for extension-created objects. > I'm not, at this point, 100% certain that that's a fully realizable > goal. But I definitely think it's insane to expect that to work > without also tracking changes in the ownership of said objects. > > Maybe forbidding ALTER OWNER on extension-owned objects isn't > such a bad idea? I think the root of my confusion, or at least one of the roots of my confusion, was whether we were talking about altering the extension or the objects within the extension. If somebody creates an extension as user nitin and afterwards changes the ownership to user lucia, then I would hope for the same pg_init_privs contents as if user lucia had created the extension in the first place. I think that might be hard to achieve in complex cases, if the extension changes the ownership of some objects during the creation process, or if some ownership has been changed afterward. But in the simple case where nothing like that happens, it sounds possible. If, on the other hand, somebody creates the extension and modifies the ownership of an object in the extension, then I agree that pg_init_privs shouldn't be updated. However, it also seems pretty clear that pg_init_privs is not recording enough state for pg_dump to mimic the ownership change, because it only records the original privileges, not the original ownership. So, can we recreate the original privileges without recreating the original ownership? It doesn't really seem like this is going to work in general. If nitin creates the extension and grants privileges to lucia, and then the ownership of the extension is changed to swara, then nitin is no longer a valid grantor. Even if we could fix that by modifying the grants to substitute swara for nitin, that could create impermissible circularities in the permissions graph. Maybe there are some scenarios where we can fix things up, but it doesn't seem possible in general. In a perfect world, the fix here is probably to have pg_init_privs or something similar record the ownership as well as the permissions, but that is not back-patchable and nobody's on the hook to fix up somebody else's feature. So what do we do? Imposing a constraint that you can't change the ownership of an extension-owned object, as you propose, seems fairly likely to break a few existing extension scripts, and also won't fix existing instances that are already broken, but maybe it's still worth considering if we don't have a better idea. Another line of thinking might be to somehow nerf pg_init_privs, or the use of it, so that we don't even try to cover this case e.g. if the ownership is changed, we nuke the pg_init_privs entry or resnap it to the current state, and the dump fails to recreate the original state but we get out from under that by declaring the case unsupported (with appropriate documentation changes, hopefully). pg_init_privs seems adequate for normal system catalog entries where the ownership shouldn't ever change from the bootstrap superuser, but extensions seem like they require more infrastructure. I think the only thing we absolutely have to fix here is the dangling ACL references. Those are a hazard, because the OID could be reused by an unrelated role, which seems like it could even give rise to security concerns. Making the whole pg_init_privs system actually work for this case case would be even better, but that seems to require filling in at least one major hole the original design, which is a lot to ask for (1) a back-patched fix or (2) a patch that someone who is not the original author has to try to write and be responsible for despite not being the one who created the problem. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> @ 2024-06-14 23:46 ` Tom Lane <[email protected]> 2024-06-15 18:40 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 0 siblings, 2 replies; 20+ messages in thread From: Tom Lane @ 2024-06-14 23:46 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Robert Haas <[email protected]> writes: > I think the only thing we absolutely have to fix here is the dangling > ACL references. Here's a draft patch that takes care of Hannu's example, and I think it fixes other potential dangling-reference scenarios too. I'm not sure whether I like the direction this is going, but I do think we may not be able to do a lot more than this for v17. The core change is to install SHARED_DEPENDENCY_INITACL entries in pg_shdepend for all references to non-pinned roles in pg_init_privs, whether they are the object's owner or not. Doing that ensures that we can't drop a role that is still referenced there, and it provides a basis for shdepDropOwned and shdepReassignOwned to take some kind of action for such references. The semantics I've implemented on top of that are: * ALTER OWNER does not touch pg_init_privs entries. * REASSIGN OWNED replaces pg_init_privs references with the new role, whether the references are as grantor or grantee. * DROP OWNED removes pg_init_privs mentions of the doomed role (as grantor or grantee), removing the pg_init_privs entry altogether if it goes to zero clauses. (This is what happened already, but only if if a SHARED_DEPENDENCY_INITACL entry existed.) I'm not terribly thrilled with this, because it's still possible to get into a state where a pg_init_privs entry is based on an owning role that's no longer the owner: you just have to use ALTER OWNER directly rather than via REASSIGN OWNED. While I don't think the backend has much problem with that, it probably will confuse pg_dump to some extent. However, such cases are going to confuse pg_dump anyway for reasons discussed upthread, namely that we don't really support dump/restore of extensions where not all the objects are owned by the extension owner. I'm content to leave that in the pile of unfinished work for now. An alternative definition could be that ALTER OWNER also replaces old role with new in pg_init_privs entries. That would reduce the surface for confusing pg_dump a little bit, but I don't think that I like it better, for two reasons: * ALTER OWNER would have to probe pg_init_acl for potential entries every time, which would be wasted work for most ALTERs. * This gets away from the notion that pg_init_privs should be a historical record of the state that existed at the end of CREATE EXTENSION. Now, maybe that notion is unworkable anyway, but I don't want to let go of it before we're sure about that. A couple of more-minor points for review: * As this stands, updateInitAclDependencies() no longer pays any attention to its ownerId argument, and in one place I depend on that to skip doing a rather expensive lookup of the current object owner. Perhaps we should remove that argument altogether, and in consequence simplify some other callers too? However, that would only help much if we were willing to revert 534287403's changes to pass the object's owner ID to recordExtensionInitPriv, which I'm hesitant to do because I suspect we'll end up wanting to record the owner ID explicitly in pg_init_privs entries. On the third hand, maybe we'll never do that, so perhaps we should revert those changes for now; some of them add nontrivial lookup costs. * In shdepReassignOwned, I refactored to move the switch on sdepForm->classid into a new subroutine. We could have left it where it is, but it would need a couple more tab stops of indentation which felt like too much. It's in the eye of the beholder though. Comments? regards, tom lane Attachments: [text/x-diff] make-reassign-owned-handle-pg_init_privs-1.patch (47.3K, ../../[email protected]/2-make-reassign-owned-handle-pg_init_privs-1.patch) download | inline diff: diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 2f52e70b48..a63cc71efa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7182,9 +7182,6 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l The referenced object (which must be a role) is mentioned in a <link linkend="catalog-pg-init-privs"><structname>pg_init_privs</structname></link> entry for the dependent object. - (A <symbol>SHARED_DEPENDENCY_INITACL</symbol> entry is not made for - the owner of the object, since the owner will have - a <symbol>SHARED_DEPENDENCY_OWNER</symbol> entry anyway.) </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c index 143876b77f..ad61e2e4a1 100644 --- a/src/backend/catalog/aclchk.c +++ b/src/backend/catalog/aclchk.c @@ -4846,6 +4846,124 @@ recordExtensionInitPrivWorker(Oid objoid, Oid classoid, int objsubid, table_close(relation, RowExclusiveLock); } +/* + * ReplaceRoleInInitPriv + * + * Used by shdepReassignOwned to replace mentions of a role in pg_init_privs. + */ +void +ReplaceRoleInInitPriv(Oid oldroleid, Oid newroleid, + Oid classid, Oid objid, int32 objsubid) +{ + Relation rel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple oldtuple; + Datum oldAclDatum; + bool isNull; + Acl *old_acl; + Acl *new_acl; + HeapTuple newtuple; + int noldmembers; + int nnewmembers; + Oid *oldmembers; + Oid *newmembers; + + /* Search for existing pg_init_privs entry for the target object. */ + rel = table_open(InitPrivsRelationId, RowExclusiveLock); + + ScanKeyInit(&key[0], + Anum_pg_init_privs_objoid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(objid)); + ScanKeyInit(&key[1], + Anum_pg_init_privs_classoid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(classid)); + ScanKeyInit(&key[2], + Anum_pg_init_privs_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(objsubid)); + + scan = systable_beginscan(rel, InitPrivsObjIndexId, true, + NULL, 3, key); + + /* There should exist only one entry or none. */ + oldtuple = systable_getnext(scan); + + if (!HeapTupleIsValid(oldtuple)) + { + /* + * Hmm, why are we here if there's no entry? But pack up and go away + * quietly. + */ + systable_endscan(scan); + table_close(rel, RowExclusiveLock); + return; + } + + /* Get a writable copy of the existing ACL. */ + oldAclDatum = heap_getattr(oldtuple, Anum_pg_init_privs_initprivs, + RelationGetDescr(rel), &isNull); + if (isNull) + { + /* Nothing needs to be changed; pack up and go away quietly. */ + systable_endscan(scan); + table_close(rel, RowExclusiveLock); + return; + } + old_acl = DatumGetAclPCopy(oldAclDatum); + + /* + * Generate new ACL. This usage of aclnewowner is a bit off-label when + * oldroleid isn't the owner; but it does the job fine. + */ + new_acl = aclnewowner(old_acl, oldroleid, newroleid); + + /* + * If we end with an empty ACL, delete the pg_init_privs entry. (That + * probably can't happen here, but we may as well cover the case.) + */ + if (new_acl == NULL || ACL_NUM(new_acl) == 0) + { + CatalogTupleDelete(rel, &oldtuple->t_self); + } + else + { + Datum values[Natts_pg_init_privs] = {0}; + bool nulls[Natts_pg_init_privs] = {0}; + bool replaces[Natts_pg_init_privs] = {0}; + + /* Update existing entry. */ + values[Anum_pg_init_privs_initprivs - 1] = PointerGetDatum(new_acl); + replaces[Anum_pg_init_privs_initprivs - 1] = true; + + newtuple = heap_modify_tuple(oldtuple, RelationGetDescr(rel), + values, nulls, replaces); + CatalogTupleUpdate(rel, &newtuple->t_self, newtuple); + } + + /* + * Update the shared dependency ACL info. We rely on the knowledge that + * updateInitAclDependencies doesn't really pay attention to its ownerId + * argument, so it doesn't matter whether that's accurate or not. + */ + noldmembers = aclmembers(old_acl, &oldmembers); + nnewmembers = aclmembers(new_acl, &newmembers); + + updateInitAclDependencies(classid, objid, objsubid, + oldroleid, + noldmembers, oldmembers, + nnewmembers, newmembers); + + systable_endscan(scan); + + /* prevent error when processing objects multiple times */ + CommandCounterIncrement(); + + table_close(rel, RowExclusiveLock); +} + /* * RemoveRoleFromInitPriv * diff --git a/src/backend/catalog/pg_shdepend.c b/src/backend/catalog/pg_shdepend.c index 20bcfd779b..d1b005d4ff 100644 --- a/src/backend/catalog/pg_shdepend.c +++ b/src/backend/catalog/pg_shdepend.c @@ -103,6 +103,9 @@ static void storeObjectDescription(StringInfo descs, ObjectAddress *object, SharedDependencyType deptype, int count); +static void shdepReassignOwned_Owner(Form_pg_shdepend sdepForm, Oid newrole); +static void shdepReassignOwned_InitAcl(Form_pg_shdepend sdepForm, + Oid oldrole, Oid newrole); /* @@ -345,10 +348,11 @@ changeDependencyOnOwner(Oid classId, Oid objectId, Oid newOwnerId) AuthIdRelationId, newOwnerId, SHARED_DEPENDENCY_ACL); - /* The same applies to SHARED_DEPENDENCY_INITACL */ - shdepDropDependency(sdepRel, classId, objectId, 0, true, - AuthIdRelationId, newOwnerId, - SHARED_DEPENDENCY_INITACL); + /* + * However, nothing need be done about SHARED_DEPENDENCY_INITACL entries, + * since those exist whether or not the role is the object's owner, and + * ALTER OWNER does not modify the underlying pg_init_privs entry. + */ table_close(sdepRel, RowExclusiveLock); } @@ -501,6 +505,8 @@ updateAclDependencies(Oid classId, Oid objectId, int32 objsubId, * * Exactly like updateAclDependencies, except we are considering a * pg_init_privs ACL for the specified object. + * + * Note: with the current semantics, ownerId isn't actually consulted. */ void updateInitAclDependencies(Oid classId, Oid objectId, int32 objsubId, @@ -542,11 +548,13 @@ updateAclDependenciesWorker(Oid classId, Oid objectId, int32 objsubId, Oid roleid = newmembers[i]; /* - * Skip the owner: he has an OWNER shdep entry instead. (This is - * not just a space optimization; it makes ALTER OWNER easier. See - * notes in changeDependencyOnOwner.) + * For SHARED_DEPENDENCY_ACL entries, skip the owner: she has an + * OWNER shdep entry instead. (This is not just a space + * optimization; it makes ALTER OWNER easier. See notes in + * changeDependencyOnOwner.) But for INITACL entries, we record + * the owner too. */ - if (roleid == ownerId) + if (deptype == SHARED_DEPENDENCY_ACL && roleid == ownerId) continue; /* Skip pinned roles; they don't need dependency entries */ @@ -563,8 +571,8 @@ updateAclDependenciesWorker(Oid classId, Oid objectId, int32 objsubId, { Oid roleid = oldmembers[i]; - /* Skip the owner, same as above */ - if (roleid == ownerId) + /* Skip the owner for ACL entries, same as above */ + if (deptype == SHARED_DEPENDENCY_ACL && roleid == ownerId) continue; /* Skip pinned roles */ @@ -1476,6 +1484,13 @@ shdepDropOwned(List *roleids, DropBehavior behavior) } break; case SHARED_DEPENDENCY_INITACL: + + /* + * Any mentions of the role that remain in pg_init_privs + * entries are just dropped. This is the same policy as + * we apply to regular ACLs. + */ + /* Shouldn't see a role grant here */ Assert(sdepForm->classid != AuthMemRelationId); RemoveRoleFromInitPriv(roleid, @@ -1577,12 +1592,8 @@ shdepReassignOwned(List *roleids, Oid newrole) sdepForm->dbid != InvalidOid) continue; - /* We leave non-owner dependencies alone */ - if (sdepForm->deptype != SHARED_DEPENDENCY_OWNER) - continue; - /* - * The various ALTER OWNER routines tend to leak memory in + * The various DDL routines called here tend to leak memory in * CurrentMemoryContext. That's not a problem when they're only * called once per command; but in this usage where we might be * touching many objects, it can amount to a serious memory leak. @@ -1593,81 +1604,23 @@ shdepReassignOwned(List *roleids, Oid newrole) ALLOCSET_DEFAULT_SIZES); oldcxt = MemoryContextSwitchTo(cxt); - /* Issue the appropriate ALTER OWNER call */ - switch (sdepForm->classid) + /* Perform the appropriate processing */ + switch (sdepForm->deptype) { - case TypeRelationId: - AlterTypeOwner_oid(sdepForm->objid, newrole, true); - break; - - case NamespaceRelationId: - AlterSchemaOwner_oid(sdepForm->objid, newrole); - break; - - case RelationRelationId: - - /* - * Pass recursing = true so that we don't fail on indexes, - * owned sequences, etc when we happen to visit them - * before their parent table. - */ - ATExecChangeOwner(sdepForm->objid, newrole, true, AccessExclusiveLock); - break; - - case DefaultAclRelationId: - - /* - * Ignore default ACLs; they should be handled by DROP - * OWNED, not REASSIGN OWNED. - */ - break; - - case UserMappingRelationId: - /* ditto */ - break; - - case ForeignServerRelationId: - AlterForeignServerOwner_oid(sdepForm->objid, newrole); - break; - - case ForeignDataWrapperRelationId: - AlterForeignDataWrapperOwner_oid(sdepForm->objid, newrole); - break; - - case EventTriggerRelationId: - AlterEventTriggerOwner_oid(sdepForm->objid, newrole); - break; - - case PublicationRelationId: - AlterPublicationOwner_oid(sdepForm->objid, newrole); + case SHARED_DEPENDENCY_OWNER: + shdepReassignOwned_Owner(sdepForm, newrole); break; - - case SubscriptionRelationId: - AlterSubscriptionOwner_oid(sdepForm->objid, newrole); + case SHARED_DEPENDENCY_INITACL: + shdepReassignOwned_InitAcl(sdepForm, roleid, newrole); break; - - /* Generic alter owner cases */ - case CollationRelationId: - case ConversionRelationId: - case OperatorRelationId: - case ProcedureRelationId: - case LanguageRelationId: - case LargeObjectRelationId: - case OperatorFamilyRelationId: - case OperatorClassRelationId: - case ExtensionRelationId: - case StatisticExtRelationId: - case TableSpaceRelationId: - case DatabaseRelationId: - case TSConfigRelationId: - case TSDictionaryRelationId: - AlterObjectOwner_internal(sdepForm->classid, - sdepForm->objid, - newrole); + case SHARED_DEPENDENCY_ACL: + case SHARED_DEPENDENCY_POLICY: + case SHARED_DEPENDENCY_TABLESPACE: + /* Nothing to do for these entry types */ break; - default: - elog(ERROR, "unexpected classid %u", sdepForm->classid); + elog(ERROR, "unrecognized dependency type: %d", + (int) sdepForm->deptype); break; } @@ -1684,3 +1637,123 @@ shdepReassignOwned(List *roleids, Oid newrole) table_close(sdepRel, RowExclusiveLock); } + +/* + * shdepReassignOwned_Owner + * + * shdepReassignOwned's processing of SHARED_DEPENDENCY_OWNER entries + */ +static void +shdepReassignOwned_Owner(Form_pg_shdepend sdepForm, Oid newrole) +{ + /* Issue the appropriate ALTER OWNER call */ + switch (sdepForm->classid) + { + case TypeRelationId: + AlterTypeOwner_oid(sdepForm->objid, newrole, true); + break; + + case NamespaceRelationId: + AlterSchemaOwner_oid(sdepForm->objid, newrole); + break; + + case RelationRelationId: + + /* + * Pass recursing = true so that we don't fail on indexes, owned + * sequences, etc when we happen to visit them before their parent + * table. + */ + ATExecChangeOwner(sdepForm->objid, newrole, true, AccessExclusiveLock); + break; + + case DefaultAclRelationId: + + /* + * Ignore default ACLs; they should be handled by DROP OWNED, not + * REASSIGN OWNED. + */ + break; + + case UserMappingRelationId: + /* ditto */ + break; + + case ForeignServerRelationId: + AlterForeignServerOwner_oid(sdepForm->objid, newrole); + break; + + case ForeignDataWrapperRelationId: + AlterForeignDataWrapperOwner_oid(sdepForm->objid, newrole); + break; + + case EventTriggerRelationId: + AlterEventTriggerOwner_oid(sdepForm->objid, newrole); + break; + + case PublicationRelationId: + AlterPublicationOwner_oid(sdepForm->objid, newrole); + break; + + case SubscriptionRelationId: + AlterSubscriptionOwner_oid(sdepForm->objid, newrole); + break; + + /* Generic alter owner cases */ + case CollationRelationId: + case ConversionRelationId: + case OperatorRelationId: + case ProcedureRelationId: + case LanguageRelationId: + case LargeObjectRelationId: + case OperatorFamilyRelationId: + case OperatorClassRelationId: + case ExtensionRelationId: + case StatisticExtRelationId: + case TableSpaceRelationId: + case DatabaseRelationId: + case TSConfigRelationId: + case TSDictionaryRelationId: + AlterObjectOwner_internal(sdepForm->classid, + sdepForm->objid, + newrole); + break; + + default: + elog(ERROR, "unexpected classid %u", sdepForm->classid); + break; + } +} + +/* + * shdepReassignOwned_InitAcl + * + * shdepReassignOwned's processing of SHARED_DEPENDENCY_INITACL entries + */ +static void +shdepReassignOwned_InitAcl(Form_pg_shdepend sdepForm, Oid oldrole, Oid newrole) +{ + /* + * Currently, REASSIGN OWNED replaces mentions of oldrole with newrole in + * pg_init_privs entries, just as it does in the object's regular ACL. + * This is less than ideal, since pg_init_privs ought to retain a + * historical record of the situation at the end of CREATE EXTENSION. + * However, there are two big stumbling blocks to doing something + * different: + * + * 1. If we don't replace the references, what is to happen if the old + * role gets dropped? (DROP OWNED's current answer is to just delete the + * pg_init_privs entry, which is surely ahistorical.) + * + * 2. It's unlikely that pg_dump will cope nicely with pg_init_privs + * entries that are based on a different owner than the object now has --- + * the more so given that pg_init_privs doesn't record the original owner + * explicitly. (This problem actually exists anyway given that a bare + * ALTER OWNER won't update pg_init_privs, but we don't need REASSIGN + * OWNED making it worse.) + */ + ReplaceRoleInInitPriv(oldrole, newrole, + sdepForm->classid, + sdepForm->objid, + sdepForm->objsubid); +} diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c index dc10b4a483..d7b39140b3 100644 --- a/src/backend/utils/adt/acl.c +++ b/src/backend/utils/adt/acl.c @@ -1091,6 +1091,12 @@ aclupdate(const Acl *old_acl, const AclItem *mod_aip, * The result is a modified copy; the input object is not changed. * * NB: caller is responsible for having detoasted the input ACL, if needed. + * + * Note: the name of this function is a bit of a misnomer, since it will + * happily make the specified role substitution whether the old role is + * really the owner of the parent object or merely mentioned in its ACL. + * But the vast majority of callers use it in connection with ALTER OWNER + * operations, so we'll keep the name. */ Acl * aclnewowner(const Acl *old_acl, Oid oldOwnerId, Oid newOwnerId) diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 7eee66f810..aca0b0b6fe 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -57,10 +57,10 @@ typedef enum DependencyType * one or the other, but not both, of these dependency types.) * * (c) a SHARED_DEPENDENCY_INITACL entry means that the referenced object is - * a role mentioned in a pg_init_privs entry for the dependent object. The - * referenced object must be a pg_authid entry. (SHARED_DEPENDENCY_INITACL - * entries are not created for the owner of an object; hence two objects may - * be linked by one or the other, but not both, of these dependency types.) + * a role mentioned in a pg_init_privs entry for the dependent object. + * The referenced object must be a pg_authid entry. (Unlike the case for + * SHARED_DEPENDENCY_ACL, we make an entry for such a role whether or not + * it is the object's owner.) * * (d) a SHARED_DEPENDENCY_POLICY entry means that the referenced object is * a role mentioned in a policy object. The referenced object must be a diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h index 1a554c6699..731d84b2a9 100644 --- a/src/include/utils/acl.h +++ b/src/include/utils/acl.h @@ -276,6 +276,8 @@ extern void aclcheck_error_type(AclResult aclerr, Oid typeOid); extern void recordExtObjInitPriv(Oid objoid, Oid classoid); extern void removeExtObjInitPriv(Oid objoid, Oid classoid); +extern void ReplaceRoleInInitPriv(Oid oldroleid, Oid newroleid, + Oid classid, Oid objid, int32 objsubid); extern void RemoveRoleFromInitPriv(Oid roleid, Oid classid, Oid objid, int32 objsubid); diff --git a/src/test/modules/test_pg_dump/expected/test_pg_dump.out b/src/test/modules/test_pg_dump/expected/test_pg_dump.out index 3cd7db97aa..3536d636d8 100644 --- a/src/test/modules/test_pg_dump/expected/test_pg_dump.out +++ b/src/test/modules/test_pg_dump/expected/test_pg_dump.out @@ -283,3 +283,308 @@ SELECT * FROM pg_init_privs WHERE privtype = 'e'; --------+----------+----------+----------+----------- (0 rows) +CREATE ROLE regress_dump_test_role; +CREATE ROLE regress_dump_test_super SUPERUSER; +SET ROLE regress_dump_test_super; +CREATE EXTENSION test_pg_dump; +RESET ROLE; +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; + obj | grantor | grantee | privilege_type | is_grantable +----------------------------------------------------+-------------------------+-------------------------+----------------+-------------- + column col1 of table regress_pg_dump_table | regress_dump_test_super | - | SELECT | f + function regress_pg_dump_schema.test_agg(smallint) | regress_dump_test_super | - | EXECUTE | f + function regress_pg_dump_schema.test_agg(smallint) | regress_dump_test_super | regress_dump_test_super | EXECUTE | f + function regress_pg_dump_schema.test_agg(smallint) | regress_dump_test_super | regress_dump_test_role | EXECUTE | f + function regress_pg_dump_schema.test_func() | regress_dump_test_super | - | EXECUTE | f + function regress_pg_dump_schema.test_func() | regress_dump_test_super | regress_dump_test_super | EXECUTE | f + function regress_pg_dump_schema.test_func() | regress_dump_test_super | regress_dump_test_role | EXECUTE | f + function wgo_then_no_access() | regress_dump_test_super | - | EXECUTE | f + function wgo_then_no_access() | regress_dump_test_super | regress_dump_test_super | EXECUTE | f + function wgo_then_no_access() | regress_dump_test_super | pg_signal_backend | EXECUTE | t + sequence regress_pg_dump_schema.test_seq | regress_dump_test_super | regress_dump_test_super | SELECT | f + sequence regress_pg_dump_schema.test_seq | regress_dump_test_super | regress_dump_test_super | UPDATE | f + sequence regress_pg_dump_schema.test_seq | regress_dump_test_super | regress_dump_test_super | USAGE | f + sequence regress_pg_dump_schema.test_seq | regress_dump_test_super | regress_dump_test_role | USAGE | f + sequence regress_pg_dump_seq | regress_dump_test_super | regress_dump_test_super | SELECT | f + sequence regress_pg_dump_seq | regress_dump_test_super | regress_dump_test_super | UPDATE | f + sequence regress_pg_dump_seq | regress_dump_test_super | regress_dump_test_super | USAGE | f + sequence regress_pg_dump_seq | regress_dump_test_super | regress_dump_test_role | USAGE | f + sequence regress_seq_dumpable | regress_dump_test_super | regress_dump_test_super | SELECT | f + sequence regress_seq_dumpable | regress_dump_test_super | regress_dump_test_super | UPDATE | f + sequence regress_seq_dumpable | regress_dump_test_super | regress_dump_test_super | USAGE | f + sequence regress_seq_dumpable | regress_dump_test_super | - | SELECT | f + sequence wgo_then_regular | regress_dump_test_super | regress_dump_test_super | SELECT | f + sequence wgo_then_regular | regress_dump_test_super | regress_dump_test_super | UPDATE | f + sequence wgo_then_regular | regress_dump_test_super | regress_dump_test_super | USAGE | f + sequence wgo_then_regular | regress_dump_test_super | pg_signal_backend | SELECT | f + sequence wgo_then_regular | regress_dump_test_super | pg_signal_backend | UPDATE | t + sequence wgo_then_regular | regress_dump_test_super | pg_signal_backend | USAGE | t + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | INSERT | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | SELECT | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | UPDATE | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | DELETE | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | TRUNCATE | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | REFERENCES | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | TRIGGER | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_super | MAINTAIN | f + table regress_pg_dump_schema.test_table | regress_dump_test_super | regress_dump_test_role | SELECT | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | INSERT | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | SELECT | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | UPDATE | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | DELETE | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | TRUNCATE | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | REFERENCES | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | TRIGGER | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_super | MAINTAIN | f + table regress_pg_dump_table | regress_dump_test_super | regress_dump_test_role | SELECT | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | INSERT | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | SELECT | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | UPDATE | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | DELETE | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | TRUNCATE | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | REFERENCES | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | TRIGGER | f + table regress_table_dumpable | regress_dump_test_super | regress_dump_test_super | MAINTAIN | f + table regress_table_dumpable | regress_dump_test_super | - | SELECT | f + type regress_pg_dump_schema.test_type | regress_dump_test_super | - | USAGE | f + type regress_pg_dump_schema.test_type | regress_dump_test_super | regress_dump_test_super | USAGE | f + type regress_pg_dump_schema.test_type | regress_dump_test_super | regress_dump_test_role | USAGE | f +(58 rows) + +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + obj | refobj | deptype +----------------------------------------------------+------------------------------+--------- + column col1 of table regress_pg_dump_table | role regress_dump_test_super | i + extension test_pg_dump | role regress_dump_test_super | o + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_role | a + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_role | i + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_super | i + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_super | o + function regress_pg_dump_schema.test_func() | role regress_dump_test_role | a + function regress_pg_dump_schema.test_func() | role regress_dump_test_role | i + function regress_pg_dump_schema.test_func() | role regress_dump_test_super | i + function regress_pg_dump_schema.test_func() | role regress_dump_test_super | o + function wgo_then_no_access() | role regress_dump_test_super | i + function wgo_then_no_access() | role regress_dump_test_super | o + schema regress_pg_dump_schema | role regress_dump_test_super | o + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_role | a + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_role | i + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_super | i + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_super | o + sequence regress_pg_dump_seq | role regress_dump_test_role | a + sequence regress_pg_dump_seq | role regress_dump_test_role | i + sequence regress_pg_dump_seq | role regress_dump_test_super | i + sequence regress_pg_dump_seq | role regress_dump_test_super | o + sequence regress_pg_dump_table_col1_seq | role regress_dump_test_super | o + sequence regress_seq_dumpable | role regress_dump_test_super | i + sequence regress_seq_dumpable | role regress_dump_test_super | o + sequence wgo_then_regular | role regress_dump_test_super | i + sequence wgo_then_regular | role regress_dump_test_super | o + table regress_pg_dump_schema.test_table | role regress_dump_test_role | a + table regress_pg_dump_schema.test_table | role regress_dump_test_role | i + table regress_pg_dump_schema.test_table | role regress_dump_test_super | i + table regress_pg_dump_schema.test_table | role regress_dump_test_super | o + table regress_pg_dump_table | role regress_dump_test_role | a + table regress_pg_dump_table | role regress_dump_test_role | i + table regress_pg_dump_table | role regress_dump_test_super | i + table regress_pg_dump_table | role regress_dump_test_super | o + table regress_table_dumpable | role regress_dump_test_super | i + table regress_table_dumpable | role regress_dump_test_super | o + type regress_pg_dump_schema.test_type | role regress_dump_test_role | a + type regress_pg_dump_schema.test_type | role regress_dump_test_role | i + type regress_pg_dump_schema.test_type | role regress_dump_test_super | i + type regress_pg_dump_schema.test_type | role regress_dump_test_super | o +(40 rows) + +REASSIGN OWNED BY regress_dump_test_super TO CURRENT_ROLE; +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; + obj | grantor | grantee | privilege_type | is_grantable +----------------------------------------------------+----------+------------------------+----------------+-------------- + column col1 of table regress_pg_dump_table | postgres | - | SELECT | f + function regress_pg_dump_schema.test_agg(smallint) | postgres | - | EXECUTE | f + function regress_pg_dump_schema.test_agg(smallint) | postgres | postgres | EXECUTE | f + function regress_pg_dump_schema.test_agg(smallint) | postgres | regress_dump_test_role | EXECUTE | f + function regress_pg_dump_schema.test_func() | postgres | - | EXECUTE | f + function regress_pg_dump_schema.test_func() | postgres | postgres | EXECUTE | f + function regress_pg_dump_schema.test_func() | postgres | regress_dump_test_role | EXECUTE | f + function wgo_then_no_access() | postgres | - | EXECUTE | f + function wgo_then_no_access() | postgres | postgres | EXECUTE | f + function wgo_then_no_access() | postgres | pg_signal_backend | EXECUTE | t + sequence regress_pg_dump_schema.test_seq | postgres | postgres | SELECT | f + sequence regress_pg_dump_schema.test_seq | postgres | postgres | UPDATE | f + sequence regress_pg_dump_schema.test_seq | postgres | postgres | USAGE | f + sequence regress_pg_dump_schema.test_seq | postgres | regress_dump_test_role | USAGE | f + sequence regress_pg_dump_seq | postgres | postgres | SELECT | f + sequence regress_pg_dump_seq | postgres | postgres | UPDATE | f + sequence regress_pg_dump_seq | postgres | postgres | USAGE | f + sequence regress_pg_dump_seq | postgres | regress_dump_test_role | USAGE | f + sequence regress_seq_dumpable | postgres | postgres | SELECT | f + sequence regress_seq_dumpable | postgres | postgres | UPDATE | f + sequence regress_seq_dumpable | postgres | postgres | USAGE | f + sequence regress_seq_dumpable | postgres | - | SELECT | f + sequence wgo_then_regular | postgres | postgres | SELECT | f + sequence wgo_then_regular | postgres | postgres | UPDATE | f + sequence wgo_then_regular | postgres | postgres | USAGE | f + sequence wgo_then_regular | postgres | pg_signal_backend | SELECT | f + sequence wgo_then_regular | postgres | pg_signal_backend | UPDATE | t + sequence wgo_then_regular | postgres | pg_signal_backend | USAGE | t + table regress_pg_dump_schema.test_table | postgres | postgres | INSERT | f + table regress_pg_dump_schema.test_table | postgres | postgres | SELECT | f + table regress_pg_dump_schema.test_table | postgres | postgres | UPDATE | f + table regress_pg_dump_schema.test_table | postgres | postgres | DELETE | f + table regress_pg_dump_schema.test_table | postgres | postgres | TRUNCATE | f + table regress_pg_dump_schema.test_table | postgres | postgres | REFERENCES | f + table regress_pg_dump_schema.test_table | postgres | postgres | TRIGGER | f + table regress_pg_dump_schema.test_table | postgres | postgres | MAINTAIN | f + table regress_pg_dump_schema.test_table | postgres | regress_dump_test_role | SELECT | f + table regress_pg_dump_table | postgres | postgres | INSERT | f + table regress_pg_dump_table | postgres | postgres | SELECT | f + table regress_pg_dump_table | postgres | postgres | UPDATE | f + table regress_pg_dump_table | postgres | postgres | DELETE | f + table regress_pg_dump_table | postgres | postgres | TRUNCATE | f + table regress_pg_dump_table | postgres | postgres | REFERENCES | f + table regress_pg_dump_table | postgres | postgres | TRIGGER | f + table regress_pg_dump_table | postgres | postgres | MAINTAIN | f + table regress_pg_dump_table | postgres | regress_dump_test_role | SELECT | f + table regress_table_dumpable | postgres | postgres | INSERT | f + table regress_table_dumpable | postgres | postgres | SELECT | f + table regress_table_dumpable | postgres | postgres | UPDATE | f + table regress_table_dumpable | postgres | postgres | DELETE | f + table regress_table_dumpable | postgres | postgres | TRUNCATE | f + table regress_table_dumpable | postgres | postgres | REFERENCES | f + table regress_table_dumpable | postgres | postgres | TRIGGER | f + table regress_table_dumpable | postgres | postgres | MAINTAIN | f + table regress_table_dumpable | postgres | - | SELECT | f + type regress_pg_dump_schema.test_type | postgres | - | USAGE | f + type regress_pg_dump_schema.test_type | postgres | postgres | USAGE | f + type regress_pg_dump_schema.test_type | postgres | regress_dump_test_role | USAGE | f +(58 rows) + +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + obj | refobj | deptype +----------------------------------------------------+-----------------------------+--------- + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_role | a + function regress_pg_dump_schema.test_agg(smallint) | role regress_dump_test_role | i + function regress_pg_dump_schema.test_func() | role regress_dump_test_role | a + function regress_pg_dump_schema.test_func() | role regress_dump_test_role | i + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_role | a + sequence regress_pg_dump_schema.test_seq | role regress_dump_test_role | i + sequence regress_pg_dump_seq | role regress_dump_test_role | a + sequence regress_pg_dump_seq | role regress_dump_test_role | i + table regress_pg_dump_schema.test_table | role regress_dump_test_role | a + table regress_pg_dump_schema.test_table | role regress_dump_test_role | i + table regress_pg_dump_table | role regress_dump_test_role | a + table regress_pg_dump_table | role regress_dump_test_role | i + type regress_pg_dump_schema.test_type | role regress_dump_test_role | a + type regress_pg_dump_schema.test_type | role regress_dump_test_role | i +(14 rows) + +DROP OWNED BY regress_dump_test_role RESTRICT; +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; + obj | grantor | grantee | privilege_type | is_grantable +----------------------------------------------------+----------+-------------------+----------------+-------------- + column col1 of table regress_pg_dump_table | postgres | - | SELECT | f + function regress_pg_dump_schema.test_agg(smallint) | postgres | - | EXECUTE | f + function regress_pg_dump_schema.test_agg(smallint) | postgres | postgres | EXECUTE | f + function regress_pg_dump_schema.test_func() | postgres | - | EXECUTE | f + function regress_pg_dump_schema.test_func() | postgres | postgres | EXECUTE | f + function wgo_then_no_access() | postgres | - | EXECUTE | f + function wgo_then_no_access() | postgres | postgres | EXECUTE | f + function wgo_then_no_access() | postgres | pg_signal_backend | EXECUTE | t + sequence regress_pg_dump_schema.test_seq | postgres | postgres | SELECT | f + sequence regress_pg_dump_schema.test_seq | postgres | postgres | UPDATE | f + sequence regress_pg_dump_schema.test_seq | postgres | postgres | USAGE | f + sequence regress_pg_dump_seq | postgres | postgres | SELECT | f + sequence regress_pg_dump_seq | postgres | postgres | UPDATE | f + sequence regress_pg_dump_seq | postgres | postgres | USAGE | f + sequence regress_seq_dumpable | postgres | postgres | SELECT | f + sequence regress_seq_dumpable | postgres | postgres | UPDATE | f + sequence regress_seq_dumpable | postgres | postgres | USAGE | f + sequence regress_seq_dumpable | postgres | - | SELECT | f + sequence wgo_then_regular | postgres | postgres | SELECT | f + sequence wgo_then_regular | postgres | postgres | UPDATE | f + sequence wgo_then_regular | postgres | postgres | USAGE | f + sequence wgo_then_regular | postgres | pg_signal_backend | SELECT | f + sequence wgo_then_regular | postgres | pg_signal_backend | UPDATE | t + sequence wgo_then_regular | postgres | pg_signal_backend | USAGE | t + table regress_pg_dump_schema.test_table | postgres | postgres | INSERT | f + table regress_pg_dump_schema.test_table | postgres | postgres | SELECT | f + table regress_pg_dump_schema.test_table | postgres | postgres | UPDATE | f + table regress_pg_dump_schema.test_table | postgres | postgres | DELETE | f + table regress_pg_dump_schema.test_table | postgres | postgres | TRUNCATE | f + table regress_pg_dump_schema.test_table | postgres | postgres | REFERENCES | f + table regress_pg_dump_schema.test_table | postgres | postgres | TRIGGER | f + table regress_pg_dump_schema.test_table | postgres | postgres | MAINTAIN | f + table regress_pg_dump_table | postgres | postgres | INSERT | f + table regress_pg_dump_table | postgres | postgres | SELECT | f + table regress_pg_dump_table | postgres | postgres | UPDATE | f + table regress_pg_dump_table | postgres | postgres | DELETE | f + table regress_pg_dump_table | postgres | postgres | TRUNCATE | f + table regress_pg_dump_table | postgres | postgres | REFERENCES | f + table regress_pg_dump_table | postgres | postgres | TRIGGER | f + table regress_pg_dump_table | postgres | postgres | MAINTAIN | f + table regress_table_dumpable | postgres | postgres | INSERT | f + table regress_table_dumpable | postgres | postgres | SELECT | f + table regress_table_dumpable | postgres | postgres | UPDATE | f + table regress_table_dumpable | postgres | postgres | DELETE | f + table regress_table_dumpable | postgres | postgres | TRUNCATE | f + table regress_table_dumpable | postgres | postgres | REFERENCES | f + table regress_table_dumpable | postgres | postgres | TRIGGER | f + table regress_table_dumpable | postgres | postgres | MAINTAIN | f + table regress_table_dumpable | postgres | - | SELECT | f + type regress_pg_dump_schema.test_type | postgres | - | USAGE | f + type regress_pg_dump_schema.test_type | postgres | postgres | USAGE | f +(51 rows) + +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + obj | refobj | deptype +-----+--------+--------- +(0 rows) + +DROP ROLE regress_dump_test_super; +DROP ROLE regress_dump_test_role; diff --git a/src/test/modules/test_pg_dump/sql/test_pg_dump.sql b/src/test/modules/test_pg_dump/sql/test_pg_dump.sql index 2110ac7163..87e66cae6e 100644 --- a/src/test/modules/test_pg_dump/sql/test_pg_dump.sql +++ b/src/test/modules/test_pg_dump/sql/test_pg_dump.sql @@ -151,3 +151,74 @@ DROP EXTENSION test_pg_dump; -- shouldn't be anything left in pg_init_privs SELECT * FROM pg_init_privs WHERE privtype = 'e'; + +CREATE ROLE regress_dump_test_role; +CREATE ROLE regress_dump_test_super SUPERUSER; + +SET ROLE regress_dump_test_super; + +CREATE EXTENSION test_pg_dump; + +RESET ROLE; + +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + +REASSIGN OWNED BY regress_dump_test_super TO CURRENT_ROLE; + +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + +DROP OWNED BY regress_dump_test_role RESTRICT; + +-- Substitute for current user's name to keep test output consistent +SELECT s.obj, + CASE WHEN a.grantor::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantor::regrole::name END, + CASE WHEN a.grantee::regrole::name = quote_ident(current_user) THEN 'postgres' + ELSE a.grantee::regrole::name END, + a.privilege_type, a.is_grantable +FROM + (SELECT pg_describe_object(classoid,objoid,objsubid) COLLATE "C" AS obj, initprivs + FROM pg_init_privs WHERE privtype = 'e' ORDER BY 1) s, + aclexplode(s.initprivs) a; +SELECT pg_describe_object(classid,objid,objsubid) COLLATE "C" AS obj, + pg_describe_object(refclassid,refobjid,0) AS refobj, + deptype + FROM pg_shdepend JOIN pg_database d ON dbid = d.oid + WHERE d.datname = current_database() + ORDER BY 1, 2, 3; + +DROP ROLE regress_dump_test_super; + +DROP ROLE regress_dump_test_role; ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> @ 2024-06-15 18:40 ` Tom Lane <[email protected]> 1 sibling, 0 replies; 20+ messages in thread From: Tom Lane @ 2024-06-15 18:40 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> I wrote: > The semantics I've implemented on top of that are: > * ALTER OWNER does not touch pg_init_privs entries. > * REASSIGN OWNED replaces pg_init_privs references with the new > role, whether the references are as grantor or grantee. > * DROP OWNED removes pg_init_privs mentions of the doomed role > (as grantor or grantee), removing the pg_init_privs entry > altogether if it goes to zero clauses. (This is what happened > already, but only if if a SHARED_DEPENDENCY_INITACL entry existed.) > I'm not terribly thrilled with this, because it's still possible > to get into a state where a pg_init_privs entry is based on > an owning role that's no longer the owner: you just have to use > ALTER OWNER directly rather than via REASSIGN OWNED. While > I don't think the backend has much problem with that, it probably > will confuse pg_dump to some extent. I poked at this some more, and I'm now moderately convinced that this is a good place to stop for v17, primarily because ALTER OWNER doesn't touch pg_init_privs in the older branches either. Thus, while we've not made things better for pg_dump, at least we've not introduced a new behavior it will have to account for in the future. I experimented with this test scenario (to set up, do "make install" in src/test/modules/test_pg_dump): ----- create role regress_dump_test_role; create user test_super superuser; create user joe; create database tpd; \c tpd test_super create extension test_pg_dump; alter function regress_pg_dump_schema.test_func() owner to joe; \df+ regress_pg_dump_schema.test_func() ----- The \df+ will correctly show that test_func() is owned by joe and has ACL | Access privileges | +------------------------------+ | =X/joe +| | joe=X/joe +| | regress_dump_test_role=X/joe | Now, if you pg_dump this database, what you get is CREATE EXTENSION IF NOT EXISTS test_pg_dump WITH SCHEMA public; ... -- -- Name: FUNCTION test_func(); Type: ACL; Schema: regress_pg_dump_schema; Owner: joe -- REVOKE ALL ON FUNCTION regress_pg_dump_schema.test_func() FROM PUBLIC; REVOKE ALL ON FUNCTION regress_pg_dump_schema.test_func() FROM test_super; REVOKE ALL ON FUNCTION regress_pg_dump_schema.test_func() FROM regress_dump_test_role; GRANT ALL ON FUNCTION regress_pg_dump_schema.test_func() TO joe; GRANT ALL ON FUNCTION regress_pg_dump_schema.test_func() TO PUBLIC; GRANT ALL ON FUNCTION regress_pg_dump_schema.test_func() TO regress_dump_test_role; So pg_dump realizes that the privileges are not what they were, but it's fairly confused about what to do about it. And it really can't get that right without better modeling (er, more than none at all) of the ownership of extension-member objects. If you restore this dump as postgres, you'll find that test_func is now owned by postgres and has ACL | Access privileges | +-----------------------------------+ | postgres=X/postgres +| | joe=X/postgres +| | =X/postgres +| | regress_dump_test_role=X/postgres | The grantees are okay, more or less, but we've totally failed to replicate the owner/grantor. But this is exactly the same as what you get if you do the experiment in v16 or before. Given the lack of complaints about that, I think it's okay to stop here for now. We've at least made REASSIGN OWNED work better. regards, tom lane ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> @ 2024-06-17 13:22 ` Daniel Gustafsson <[email protected]> 2024-06-17 14:56 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 1 sibling, 1 reply; 20+ messages in thread From: Daniel Gustafsson @ 2024-06-17 13:22 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> > On 15 Jun 2024, at 01:46, Tom Lane <[email protected]> wrote: > > Robert Haas <[email protected]> writes: >> I think the only thing we absolutely have to fix here is the dangling >> ACL references. > > Here's a draft patch that takes care of Hannu's example, and I think > it fixes other potential dangling-reference scenarios too. > > I'm not sure whether I like the direction this is going, but I do > think we may not be able to do a lot more than this for v17. Agreed. > The core change is to install SHARED_DEPENDENCY_INITACL entries in > pg_shdepend for all references to non-pinned roles in pg_init_privs, > whether they are the object's owner or not. Doing that ensures that > we can't drop a role that is still referenced there, and it provides > a basis for shdepDropOwned and shdepReassignOwned to take some kind > of action for such references. I wonder if this will break any tools/scripts in prod which relies on the previous (faulty) behaviour. It will be interesting to see if anything shows up on -bugs. Off the cuff it seems like a good idea judging by where we are and what we can fix with it. > I'm not terribly thrilled with this, because it's still possible > to get into a state where a pg_init_privs entry is based on > an owning role that's no longer the owner: you just have to use > ALTER OWNER directly rather than via REASSIGN OWNED. While > I don't think the backend has much problem with that, it probably > will confuse pg_dump to some extent. However, such cases are > going to confuse pg_dump anyway for reasons discussed upthread, > namely that we don't really support dump/restore of extensions > where not all the objects are owned by the extension owner. > I'm content to leave that in the pile of unfinished work for now. +1 > An alternative definition could be that ALTER OWNER also replaces > old role with new in pg_init_privs entries. That would reduce > the surface for confusing pg_dump a little bit, but I don't think > that I like it better, for two reasons: > > * ALTER OWNER would have to probe pg_init_acl for potential > entries every time, which would be wasted work for most ALTERs. Unless it would magically fix all the pg_dump problems I'd prefer to avoid this. > * As this stands, updateInitAclDependencies() no longer pays any > attention to its ownerId argument, and in one place I depend on > that to skip doing a rather expensive lookup of the current object > owner. Perhaps we should remove that argument altogether, and > in consequence simplify some other callers too? However, that > would only help much if we were willing to revert 534287403's > changes to pass the object's owner ID to recordExtensionInitPriv, > which I'm hesitant to do because I suspect we'll end up wanting > to record the owner ID explicitly in pg_init_privs entries. > On the third hand, maybe we'll never do that, so perhaps we should > revert those changes for now; some of them add nontrivial lookup > costs. I wonder if it's worth reverting passing the owner ID for v17 and revisiting that in 18 if we work on recording the ID. Shaving a few catalog lookups is generally worthwhile, doing them without needing the result for the next five years might bite us. Re-reading 534287403 I wonder about this hunk in RemoveRoleFromInitPriv: + if (!isNull) + old_acl = DatumGetAclPCopy(oldAclDatum); + else + old_acl = NULL; /* this case shouldn't happen, probably */ I wonder if we should Assert() on old_acl being NULL? I can't imagine a case where it should legitimately be that and catching such in development might be useful for catching stray bugs? > * In shdepReassignOwned, I refactored to move the switch on > sdepForm->classid into a new subroutine. We could have left > it where it is, but it would need a couple more tab stops of > indentation which felt like too much. It's in the eye of > the beholder though. I prefer the new way. +1 on going ahead with this patch. There is more work to do but I agree that this about all that makes sense in v17 at this point. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> @ 2024-06-17 14:56 ` Tom Lane <[email protected]> 2024-06-17 16:37 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 0 siblings, 1 reply; 20+ messages in thread From: Tom Lane @ 2024-06-17 14:56 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; +Cc: Robert Haas <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Daniel Gustafsson <[email protected]> writes: > On 15 Jun 2024, at 01:46, Tom Lane <[email protected]> wrote: >> The core change is to install SHARED_DEPENDENCY_INITACL entries in >> pg_shdepend for all references to non-pinned roles in pg_init_privs, >> whether they are the object's owner or not. Doing that ensures that >> we can't drop a role that is still referenced there, and it provides >> a basis for shdepDropOwned and shdepReassignOwned to take some kind >> of action for such references. > I wonder if this will break any tools/scripts in prod which relies on the > previous (faulty) behaviour. It will be interesting to see if anything shows > up on -bugs. Off the cuff it seems like a good idea judging by where we are > and what we can fix with it. Considering that SHARED_DEPENDENCY_INITACL has existed for less than two months, it's hard to believe that any outside code has grown any dependencies on it, much less that it couldn't be adjusted readily. > I wonder if it's worth reverting passing the owner ID for v17 and revisiting > that in 18 if we work on recording the ID. Shaving a few catalog lookups is > generally worthwhile, doing them without needing the result for the next five > years might bite us. Yeah, that was the direction I was leaning in, too. I'll commit the revert of that separately, so that un-reverting it shouldn't be too painful if we eventually decide to do so. > Re-reading 534287403 I wonder about this hunk in RemoveRoleFromInitPriv: > + if (!isNull) > + old_acl = DatumGetAclPCopy(oldAclDatum); > + else > + old_acl = NULL; /* this case shouldn't happen, probably */ > I wonder if we should Assert() on old_acl being NULL? I can't imagine a case > where it should legitimately be that and catching such in development might be > useful for catching stray bugs? Hmm, yeah. I was trying to be agnostic about whether it's okay for a pg_init_privs ACL to be NULL ... but I can't imagine a real use for that either, and the new patch does add some code that's effectively assuming it isn't. Agreed, let's be uniform about insisting !isNull. > +1 on going ahead with this patch. There is more work to do but I agree that > this about all that makes sense in v17 at this point. Thanks for reviewing! regards, tom lane ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-17 14:56 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> @ 2024-06-17 16:37 ` Daniel Gustafsson <[email protected]> 2024-06-20 10:14 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> 0 siblings, 1 reply; 20+ messages in thread From: Daniel Gustafsson @ 2024-06-17 16:37 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Hannu Krosing <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> > On 17 Jun 2024, at 16:56, Tom Lane <[email protected]> wrote: > Daniel Gustafsson <[email protected]> writes: >> I wonder if this will break any tools/scripts in prod which relies on the >> previous (faulty) behaviour. It will be interesting to see if anything shows >> up on -bugs. Off the cuff it seems like a good idea judging by where we are >> and what we can fix with it. > > Considering that SHARED_DEPENDENCY_INITACL has existed for less than > two months, it's hard to believe that any outside code has grown any > dependencies on it, much less that it couldn't be adjusted readily. Doh, I was thinking about it backwards, clearly not a worry =) >> I wonder if it's worth reverting passing the owner ID for v17 and revisiting >> that in 18 if we work on recording the ID. Shaving a few catalog lookups is >> generally worthwhile, doing them without needing the result for the next five >> years might bite us. > > Yeah, that was the direction I was leaning in, too. I'll commit the > revert of that separately, so that un-reverting it shouldn't be too > painful if we eventually decide to do so. Sounds good. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-17 14:56 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 16:37 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> @ 2024-06-20 10:14 ` Hannu Krosing <[email protected]> 2024-06-20 15:35 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 0 siblings, 1 reply; 20+ messages in thread From: Hannu Krosing @ 2024-06-20 10:14 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Robert Haas <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Hi Tom, Is there anything that could be back-patched with reasonable effort ? -- Hannu On Mon, Jun 17, 2024 at 6:37 PM Daniel Gustafsson <[email protected]> wrote: > > > On 17 Jun 2024, at 16:56, Tom Lane <[email protected]> wrote: > > Daniel Gustafsson <[email protected]> writes: > > >> I wonder if this will break any tools/scripts in prod which relies on the > >> previous (faulty) behaviour. It will be interesting to see if anything shows > >> up on -bugs. Off the cuff it seems like a good idea judging by where we are > >> and what we can fix with it. > > > > Considering that SHARED_DEPENDENCY_INITACL has existed for less than > > two months, it's hard to believe that any outside code has grown any > > dependencies on it, much less that it couldn't be adjusted readily. > > Doh, I was thinking about it backwards, clearly not a worry =) > > >> I wonder if it's worth reverting passing the owner ID for v17 and revisiting > >> that in 18 if we work on recording the ID. Shaving a few catalog lookups is > >> generally worthwhile, doing them without needing the result for the next five > >> years might bite us. > > > > Yeah, that was the direction I was leaning in, too. I'll commit the > > revert of that separately, so that un-reverting it shouldn't be too > > painful if we eventually decide to do so. > > Sounds good. > > -- > Daniel Gustafsson > ^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: DROP OWNED BY fails to clean out pg_init_privs grants 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-17 14:56 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 16:37 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-20 10:14 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> @ 2024-06-20 15:35 ` Tom Lane <[email protected]> 0 siblings, 0 replies; 20+ messages in thread From: Tom Lane @ 2024-06-20 15:35 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Robert Haas <[email protected]>; David G. Johnston <[email protected]>; PostgreSQL Hackers <[email protected]>; Stephen Frost <[email protected]>; Andres Freund <[email protected]>; Noah Misch <[email protected]> Hannu Krosing <[email protected]> writes: > Is there anything that could be back-patched with reasonable effort ? Afraid not. The whole thing is dependent on pg_shdepend entries that won't exist in older branches. regards, tom lane ^ permalink raw reply [nested|flat] 20+ messages in thread
end of thread, other threads:[~2024-06-20 15:35 UTC | newest] Thread overview: 20+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-11-12 14:32 Re: Changes to error handling for background worker initialization? Jeremy Finzel <[email protected]> 2019-05-28 18:16 [PATCH v18 3/6] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH v17 4/7] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH 4/6] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH v16 4/7] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH v15 4/7] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH v14 4/7] Track collation versions for indexes. Thomas Munro <[email protected]> 2019-05-28 18:16 [PATCH v16 4/7] Track collation versions for indexes. Thomas Munro <[email protected]> 2023-07-26 19:31 [PATCH 1/4] drop has oids field instead of having static values Pierre Ducroquet <[email protected]> 2024-05-24 20:00 Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-05-25 14:09 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> 2024-05-25 14:47 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-04 14:16 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Robert Haas <[email protected]> 2024-06-14 23:46 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-15 18:40 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 13:22 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-17 14:56 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]> 2024-06-17 16:37 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Daniel Gustafsson <[email protected]> 2024-06-20 10:14 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Hannu Krosing <[email protected]> 2024-06-20 15:35 ` Re: DROP OWNED BY fails to clean out pg_init_privs grants Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox