public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v30 4/6] Track collation versions for indexes. 11+ messages / 5 participants [nested] [flat]
* [PATCH v30 4/6] Track collation versions for indexes. @ 2019-05-28 18:16 Thomas Munro <[email protected]> 0 siblings, 0 replies; 11+ messages in thread From: Thomas Munro @ 2019-05-28 18:16 UTC (permalink / raw) Record the current version of dependent collations in pg_depend when creating or rebuilding an index. The version is checked against the current version whenever we call get_relation_info for an index or open the parent table during non-full VACUUM or ANALYZE. Warn that the index may be corrupted if the versions don't match. Add a new flag is added to RelationData to record that the check has already been done to avoid repetition. Thanks to Doug Doole, Peter Eisentraut, Christoph Berg, Laurenz Albe, Michael Paquier, Robert Haas, Tom Lane and others for design discussion. Author: Thomas Munro <[email protected]> Author: Julien Rouhaud <[email protected]> Reviewed-by: Peter Eisentraut <[email protected]> Reviewed-by: Laurenz Albe <[email protected]> Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/func.sgml | 3 +- doc/src/sgml/ref/pgupgrade.sgml | 18 + doc/src/sgml/ref/reindex.sgml | 6 + src/backend/catalog/dependency.c | 219 +++++++++++-- src/backend/catalog/heap.c | 7 +- src/backend/catalog/index.c | 307 +++++++++++++++++- src/backend/catalog/pg_constraint.c | 2 +- src/backend/catalog/pg_depend.c | 262 ++++++++++++++- src/backend/catalog/pg_type.c | 69 ++++ src/backend/commands/collationcmds.c | 22 +- src/backend/commands/vacuum.c | 32 ++ src/backend/optimizer/util/plancat.c | 9 + src/backend/utils/adt/pg_locale.c | 52 ++- src/backend/utils/adt/pg_upgrade_support.c | 25 ++ src/backend/utils/cache/relcache.c | 2 + src/bin/pg_dump/Makefile | 2 + src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 186 ++++++++++- src/bin/pg_dump/pg_dump.h | 3 + src/bin/pg_dump/t/002_pg_dump.pl | 248 ++++++++++---- src/bin/pg_upgrade/dump.c | 4 +- src/bin/pg_upgrade/option.c | 7 + src/bin/pg_upgrade/pg_upgrade.h | 2 + src/include/catalog/dependency.h | 31 +- src/include/catalog/index.h | 7 + src/include/catalog/pg_proc.dat | 4 + src/include/catalog/pg_type.h | 2 + src/include/utils/pg_locale.h | 2 +- src/include/utils/rel.h | 1 + src/test/Makefile | 5 +- src/test/locale/.gitignore | 1 + src/test/locale/Makefile | 7 + src/test/locale/t/001_index.pl | 76 +++++ src/test/perl/PostgresNode.pm | 6 +- .../regress/expected/collate.icu.utf8.out | 187 +++++++++++ src/test/regress/expected/create_index.out | 8 +- src/test/regress/sql/collate.icu.utf8.sql | 126 +++++++ 37 files changed, 1803 insertions(+), 148 deletions(-) create mode 100644 src/test/locale/t/001_index.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 1f33f99040..86f2e16533 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -25462,7 +25462,8 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); </para> <para> Returns the actual version of the collation object as it is currently - installed in the operating system. + installed in the operating system. An empty string is returned if the + version is unknown. </para></entry> </row> diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index b59c5697a3..527c847e27 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -215,6 +215,24 @@ PostgreSQL documentation </listitem> </varlistentry> + <varlistentry> + <term><option>--collation-binary-compatible</option></term> + <listitem> + <para> + After upgrading from a release of <productname>PostgreSQL</productname> + before 14, warnings may be reported when collation-dependent + indexes are first accessed. This is because + <application>pg_upgrade</application> records the collation versions + as unknown, so <productname>PostgreSQL</productname> considers the + indexes to be potentially corrupted. If you're certain that the + collation definitions used by your operating system or ICU haven't + changed since all indexes were created, you can use this flag + to record that the indexes match the currently installed collations. + Otherwise, see <xref linkend="sql-reindex"/>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-?</option></term> <term><option>--help</option></term> diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index fa43e3a972..ef047a8945 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -39,6 +39,12 @@ REINDEX [ ( <replaceable class="parameter">option</replaceable> [, ...] ) ] { IN <itemizedlist> <listitem> + <para> + The index depends on the sort order of a collation, and the definition + of the collation has changed. This can cause an index to fail to + find a key that is present in the index. On some systems, collation + versions are tracked in order to generate warnings when this happens. + </para> <para> An index has become corrupted, and no longer contains valid data. Although in theory this should never happen, in diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c index 1a927377e7..c93fdbd7d5 100644 --- a/src/backend/catalog/dependency.c +++ b/src/backend/catalog/dependency.c @@ -76,6 +76,7 @@ #include "rewrite/rewriteRemove.h" #include "storage/lmgr.h" #include "utils/acl.h" +#include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/guc.h" #include "utils/lsyscache.h" @@ -136,6 +137,9 @@ typedef struct { ObjectAddresses *addrs; /* addresses being accumulated */ List *rtables; /* list of rangetables to resolve Vars */ + bool track_version; /* whether caller asked to track dependency + * versions */ + NodeTag type; /* nodetag of the current node */ } find_expr_references_context; /* @@ -434,6 +438,81 @@ performMultipleDeletions(const ObjectAddresses *objects, table_close(depRel, RowExclusiveLock); } +/* + * Call a function for all objects that depend on 'object'. If the function + * returns a non-NULL pointer to a new version string, update the version. + */ +void +visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + ObjectAddress otherObject; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(object->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(object->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + char *cur_version, + *new_version; + Datum depversion; + bool isnull; + + otherObject.classId = foundDep->refclassid; + otherObject.objectId = foundDep->refobjid; + otherObject.objectSubId = foundDep->refobjsubid; + + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + + new_version = callback(&otherObject, cur_version, userdata); + if (new_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(new_version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); +} + /* * findDependentObjects - find all objects that depend on 'object' * @@ -1566,6 +1645,37 @@ ReleaseDeletionLock(const ObjectAddress *object) AccessExclusiveLock); } +/* + * Given a list of collations, record a dependency on its underlying collation + * version. + */ +void +recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version) +{ + ObjectAddresses *addrs; + ListCell *lc; + + if (list_length(collations) == 0) + return; + + addrs = new_object_addresses(); + + foreach(lc, collations) + { + ObjectAddress referenced; + + ObjectAddressSet(referenced, CollationRelationId, lfirst_oid(lc)); + + add_exact_object_address(&referenced, addrs); + } + + eliminate_duplicate_dependencies(addrs); + recordMultipleDependencies(myself, addrs->refs, addrs->numrefs, + DEPENDENCY_NORMAL, record_version); +} + /* * recordDependencyOnExpr - find expression dependencies * @@ -1588,6 +1698,10 @@ recordDependencyOnExpr(const ObjectAddress *depender, find_expr_references_context context; context.addrs = new_object_addresses(); + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* Set up interpretation for Vars at varlevelsup = 0 */ context.rtables = list_make1(rtable); @@ -1602,8 +1716,8 @@ recordDependencyOnExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + false); free_object_addresses(context.addrs); } @@ -1630,12 +1744,18 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self) + bool reverse_self, + bool track_version) { find_expr_references_context context; RangeTblEntry rte; context.addrs = new_object_addresses(); + context.track_version = track_version; + if (expr) + context.type = expr->type; + else + context.type = T_Invalid; /* We gin up a rather bogus rangetable list to handle Vars */ MemSet(&rte, 0, sizeof(rte)); @@ -1691,8 +1811,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, self_addrs->refs, self_addrs->numrefs, - NULL, - self_behavior); + self_behavior, + track_version); else { /* Can't use recordMultipleDependencies, so do it the hard way */ @@ -1713,8 +1833,8 @@ recordDependencyOnSingleRelExpr(const ObjectAddress *depender, recordMultipleDependencies(depender, context.addrs->refs, context.addrs->numrefs, - NULL, - behavior); + behavior, + track_version); free_object_addresses(context.addrs); } @@ -1736,8 +1856,13 @@ static bool find_expr_references_walker(Node *node, find_expr_references_context *context) { + NodeTag parent_type = context->type; + if (node == NULL) return false; + + context->type = node->type; + if (IsA(node, Var)) { Var *var = (Var *) node; @@ -1770,6 +1895,49 @@ find_expr_references_walker(Node *node, /* If it's a plain relation, reference this column */ add_object_address(OCLASS_CLASS, rte->relid, var->varattno, context->addrs); + + /* + * Record collations from the type itself, or underlying in case + * of complex type. Note that if the direct parent is a + * CollateExpr node, there's no need to record the type underlying + * collation if any. A dependency already exists for the owning + * relation, and a change in the collation sort order wouldn't + * cause any harm as the collation isn't used at all in such case. + */ + if (parent_type != T_CollateExpr) + { + /* type's collation if valid */ + if (OidIsValid(var->varcollid)) + { + add_object_address(OCLASS_COLLATION, var->varcollid, 0, + context->addrs); + } + + /* + * otherwise, it may be a composite type having underlying + * collations + */ + else if (var->vartype >= FirstNormalObjectId) + { + List *collations = NIL; + ListCell *lc; + + collations = GetTypeCollations(var->vartype, false); + + foreach(lc, collations) + { + Oid coll = lfirst_oid(lc); + + if (OidIsValid(coll) && + (coll != DEFAULT_COLLATION_OID || + context->track_version) + ) + add_object_address(OCLASS_COLLATION, + lfirst_oid(lc), 0, + context->addrs); + } + } + } } /* @@ -1795,10 +1963,12 @@ find_expr_references_walker(Node *node, * We must also depend on the constant's collation: it could be * different from the datatype's, if a CollateExpr was const-folded to * a simple constant. However we can save work in the most common - * case where the collation is "default", since we know that's pinned. + * case where the collation is "default", since we know that's pinned, + * if the caller didn't ask to track dependency versions. */ if (OidIsValid(con->constcollid) && - con->constcollid != DEFAULT_COLLATION_OID) + (con->constcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, con->constcollid, 0, context->addrs); @@ -1888,7 +2058,8 @@ find_expr_references_walker(Node *node, context->addrs); /* and its collation, just as for Consts */ if (OidIsValid(param->paramcollid) && - param->paramcollid != DEFAULT_COLLATION_OID) + (param->paramcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, param->paramcollid, 0, context->addrs); } @@ -1976,7 +2147,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(fselect->resultcollid) && - fselect->resultcollid != DEFAULT_COLLATION_OID) + (fselect->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, fselect->resultcollid, 0, context->addrs); } @@ -2007,7 +2179,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(relab->resultcollid) && - relab->resultcollid != DEFAULT_COLLATION_OID) + (relab->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, relab->resultcollid, 0, context->addrs); } @@ -2020,7 +2193,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(iocoerce->resultcollid) && - iocoerce->resultcollid != DEFAULT_COLLATION_OID) + (iocoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, iocoerce->resultcollid, 0, context->addrs); } @@ -2033,7 +2207,8 @@ find_expr_references_walker(Node *node, context->addrs); /* the collation might not be referenced anywhere else, either */ if (OidIsValid(acoerce->resultcollid) && - acoerce->resultcollid != DEFAULT_COLLATION_OID) + (acoerce->resultcollid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, acoerce->resultcollid, 0, context->addrs); /* fall through to examine arguments */ @@ -2122,7 +2297,8 @@ find_expr_references_walker(Node *node, add_object_address(OCLASS_PROC, wc->endInRangeFunc, 0, context->addrs); if (OidIsValid(wc->inRangeColl) && - wc->inRangeColl != DEFAULT_COLLATION_OID) + (wc->inRangeColl != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, wc->inRangeColl, 0, context->addrs); /* fall through to examine substructure */ @@ -2267,7 +2443,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2289,7 +2467,9 @@ find_expr_references_walker(Node *node, { Oid collid = lfirst_oid(ct); - if (OidIsValid(collid) && collid != DEFAULT_COLLATION_OID) + if (OidIsValid(collid) && + (collid != DEFAULT_COLLATION_OID || + context->track_version)) add_object_address(OCLASS_COLLATION, collid, 0, context->addrs); } @@ -2685,8 +2865,9 @@ record_object_address_dependencies(const ObjectAddress *depender, { eliminate_duplicate_dependencies(referenced); recordMultipleDependencies(depender, - referenced->refs, referenced->numrefs, NULL, - behavior); + referenced->refs, referenced->numrefs, + behavior, + false); } /* diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 67144aa3c9..fc5140daa6 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2336,7 +2336,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&colobject, expr, RelationGetRelid(rel), DEPENDENCY_AUTO, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, false); } else { @@ -2346,7 +2346,7 @@ StoreAttrDefault(Relation rel, AttrNumber attnum, */ recordDependencyOnSingleRelExpr(&defobject, expr, RelationGetRelid(rel), DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, false); } /* @@ -3665,7 +3665,8 @@ StorePartitionKey(Relation rel, RelationGetRelid(rel), DEPENDENCY_NORMAL, DEPENDENCY_INTERNAL, - true /* reverse the self-deps */ ); + true /* reverse the self-deps */ , + false /* don't track versions */ ); /* * We must invalidate the relcache so that the next diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 0974f3e23a..4c8a221e0d 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -53,6 +53,7 @@ #include "catalog/pg_trigger.h" #include "catalog/pg_type.h" #include "catalog/storage.h" +#include "commands/defrem.h" #include "commands/event_trigger.h" #include "commands/progress.h" #include "commands/tablecmds.h" @@ -75,6 +76,7 @@ #include "utils/guc.h" #include "utils/inval.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/memutils.h" #include "utils/pg_rusage.h" #include "utils/rel.h" @@ -1020,6 +1022,11 @@ index_create(Relation heapRelation, ObjectAddress myself, referenced; ObjectAddresses *addrs; + ListCell *lc; + List *colls = NIL, + *colls_pattern_ops = NIL, + *determ_colls = NIL, + *nondeterm_colls = NIL; ObjectAddressSet(myself, RelationRelationId, indexRelationId); @@ -1106,20 +1113,122 @@ index_create(Relation heapRelation, /* placeholder for normal dependencies */ addrs = new_object_addresses(); - /* Store dependency on collations */ - - /* The default collation is pinned, so don't bother recording it */ + /* First, get all dependencies on collations for all index keys. */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { - if (OidIsValid(collationObjectId[i]) && - collationObjectId[i] != DEFAULT_COLLATION_OID) + Oid colloid = collationObjectId[i]; + + if (OidIsValid(colloid)) { - ObjectAddressSet(referenced, CollationRelationId, - collationObjectId[i]); - add_exact_object_address(&referenced, addrs); + Oid opclass = classObjectId[i]; + + /* + * The *_pattern_ops opclasses are special, as they're known + * to not rely on the collation sort order. + */ + if (opclass == TEXT_BTREE_PATTERN_OPS_OID || + opclass == VARCHAR_BTREE_PATTERN_OPS_OID || + opclass == BPCHAR_BTREE_PATTERN_OPS_OID) + colls_pattern_ops = lappend_oid(colls_pattern_ops, colloid); + else + colls = lappend_oid(colls, colloid); + } + /* + * Required for e.g. custom types having multiple underlying + * collations. Note that none of the *_pattern_ops can be used in + * such constructs. + */ + else + { + Form_pg_attribute att = TupleDescAttr(indexTupDesc, i); + + Assert(i < indexTupDesc->natts); + + colls = list_concat(colls, + GetTypeCollations(att->atttypid, false)); } } + /* Check if any collation exist for both a *_pattern_ops opclass and + * another one. In that case, we have to create a single dependency + * with version tracked. + */ + if (colls_pattern_ops != NIL && colls != NIL) + colls_pattern_ops = list_difference_oid(colls_pattern_ops, colls); + + /* + * Record the dependencies for collation declares with any of the + * *_pattern_ops opclass, without version tracking. + */ + if (colls_pattern_ops != NIL) + { + recordDependencyOnCollations(&myself, colls_pattern_ops, false); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * Then split the dependencies for collations that were not declared + * with any of the *_pattern_ops opclass on whether they're + * deterministic or not, removing any duplicates. + */ + foreach(lc, colls) + { + Oid c = lfirst_oid(lc); + + if (!OidIsValid(c)) + continue; + + if (get_collation_isdeterministic(c)) + determ_colls = lappend_oid(determ_colls, c); + else + nondeterm_colls = lappend_oid(nondeterm_colls, c); + } + + /* + * For deterministic collation, we always track the dependency on the + * collation but only track the version if the AM relies on a stable + * ordering. + */ + if (determ_colls != NIL) + { + IndexAmRoutine *routine; + bool track_version; + + routine = GetIndexAmRoutineByAmId(accessMethodObjectId, false); + track_version = !routine->amnostablecollorder; + + recordDependencyOnCollations(&myself, determ_colls, track_version); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + + /* + * We always record the version for dependency on non-deterministic + * collations. + */ + if (nondeterm_colls != NIL) + { + recordDependencyOnCollations(&myself, nondeterm_colls, true); + + /* + * Advance the command counter so that later calls to + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); + } + /* Store dependency on operator classes */ for (i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) { @@ -1133,21 +1242,32 @@ index_create(Relation heapRelation, /* Store dependencies on anything mentioned in index expressions */ if (indexInfo->ii_Expressions) { + /* + * recordDependencyOnSingleRelExpr gets rid of duplicated entries + */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Expressions, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); + + /* + * Advance the command counter so that later + * recordMultipleDependencies calls can see the newly-entered + * pg_depend catalog tuples for the index. + */ + CommandCounterIncrement(); } /* Store dependencies on anything mentioned in predicate */ if (indexInfo->ii_Predicate) { + /* recordDependencyOnSingleRelExpr get rid of duplicated entries */ recordDependencyOnSingleRelExpr(&myself, (Node *) indexInfo->ii_Predicate, heapRelationId, DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + DEPENDENCY_AUTO, false, true); } } else @@ -1226,6 +1346,108 @@ index_create(Relation heapRelation, return indexRelationId; } +/* index_check_collation_version + * Raise a warning if the recorded and current collation version don't + * match. +*/ +static char * +index_check_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + Oid relid = *(Oid *) userdata; + char *current_version; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* Compare with the current version. */ + current_version = get_collation_version_for_oid(otherObject->objectId); + + /* XXX should we warn about "disappearing" versions? */ + if (current_version) + { + /* + * We now support versioning for the underlying collation library on + * this system, or previous version is unknown. + */ + if (!version || (strcmp(version, "") == 0 && strcmp(current_version, + "") != 0)) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" with an unknown version, and the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + else if (strcmp(current_version, version) != 0) + { + ereport(WARNING, + (errmsg("index \"%s\" depends on collation \"%s\" version \"%s\", but the current version is \"%s\"", + get_rel_name(relid), + get_collation_name(otherObject->objectId), + version, + current_version), + errdetail("The index may be corrupted due to changes in sort order."), + errhint("REINDEX to avoid the risk of corruption."))); + } + } + + return NULL; +} + +/* index_check_collation_versions + * Check the collation version for all dependencies on the given object. + */ +void +index_check_collation_versions(Oid relid) +{ + ObjectAddress object; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_check_collation_version, &relid); +} + +/* index_update_collation_version + * Return the current collation version for the given object. + */ +static char * +index_update_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + char *current_version = (char *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + current_version = get_collation_version_for_oid(otherObject->objectId); + return current_version; +} + +/* index_update_collation_versions + * Record the current collation versions of all dependencies on the given + * object. + */ +void +index_update_collation_versions(Oid relid) +{ + ObjectAddress object; + NameData current_version; + + object.classId = RelationRelationId; + object.objectId = relid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_update_collation_version, + ¤t_version); +} + /* * index_concurrently_create_copy * @@ -3000,6 +3222,68 @@ index_build(Relation heapRelation, SetUserIdAndSecContext(save_userid, save_sec_context); } +static char * +index_force_collation_version(const ObjectAddress *otherObject, + const char *version, + void *userdata) +{ + NewCollationVersionDependency *forced_dependency; + + forced_dependency = (NewCollationVersionDependency *) userdata; + + /* We only care about dependencies on collations. */ + if (otherObject->classId != CollationRelationId) + return NULL; + + /* + * We only care about dependencies on a specific collation if a valid Oid + * was given. + */ + if (OidIsValid(forced_dependency->oid) && + otherObject->objectId != forced_dependency->oid) + return NULL; + + return forced_dependency->version; +} + +/* index_force_collation_versions + * + * Override collation version dependencies for the given index. If no + * collation identifier is specified, all dependent collation should be + * reset to an unknown version dependency, and no version should be provided + * either. + */ +void +index_force_collation_versions(Oid indexid, Oid coll, char *version) +{ + Relation index; + ObjectAddress object; + NewCollationVersionDependency forced_dependency; + + Assert(version); + + index = relation_open(indexid, AccessExclusiveLock); + + if (index->rd_rel->relkind != RELKIND_INDEX) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + (errmsg("\"%s\" is not an index", get_rel_name(indexid))))); + + forced_dependency.oid = InvalidOid; + forced_dependency.version = version; + + object.classId = RelationRelationId; + object.objectId = indexid; + object.objectSubId = 0; + visitDependentObjects(&object, &index_force_collation_version, + &forced_dependency); + + /* Invalidate the index relcache */ + CacheInvalidateRelcache(index); + + relation_close(index, NoLock); +} + /* * IndexCheckExclusion - verify that a new exclusion constraint is satisfied * @@ -3635,6 +3919,9 @@ reindex_index(Oid indexId, bool skip_constraint_checks, char persistence, /* Close rels, but keep locks */ index_close(iRel, NoLock); table_close(heapRelation, NoLock); + + /* Record the current versions of all depended-on collations. */ + index_update_collation_versions(indexId); } /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index 0d70cb0c3c..93774c9d21 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -362,7 +362,7 @@ CreateConstraintEntry(const char *constraintName, */ recordDependencyOnSingleRelExpr(&conobject, conExpr, relId, DEPENDENCY_NORMAL, - DEPENDENCY_NORMAL, false); + DEPENDENCY_NORMAL, false, true); } /* Post creation hook for new constraint */ diff --git a/src/backend/catalog/pg_depend.c b/src/backend/catalog/pg_depend.c index 09c30b13e8..89c866cb04 100644 --- a/src/backend/catalog/pg_depend.c +++ b/src/backend/catalog/pg_depend.c @@ -19,6 +19,7 @@ #include "access/table.h" #include "catalog/dependency.h" #include "catalog/indexing.h" +#include "catalog/pg_collation.h" #include "catalog/pg_constraint.h" #include "catalog/pg_depend.h" #include "catalog/pg_extension.h" @@ -27,9 +28,15 @@ #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/lsyscache.h" +#include "utils/pg_locale.h" #include "utils/rel.h" +static bool dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced, + char *version); +static char *getDependencyVersion(const ObjectAddress *depender, + const ObjectAddress *referenced); static bool isObjectPinned(const ObjectAddress *object, Relation rel); @@ -45,19 +52,24 @@ recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior) { - recordMultipleDependencies(depender, referenced, 1, NULL, behavior); + recordMultipleDependencies(depender, referenced, 1, behavior, false); } /* * Record multiple dependencies (of the same kind) for a single dependent * object. This has a little less overhead than recording each separately. + * + * If track_version is true, then a record could be added even if the referenced + * dependency is pinned, and the dependency version will be retrieved according + * to the referenced object kind. + * For now, only collation version is supported. */ void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior) + DependencyType behavior, + bool track_version) { Relation dependDesc; CatalogIndexState indstate; @@ -66,6 +78,7 @@ recordMultipleDependencies(const ObjectAddress *depender, max_slots, slot_init_count, slot_stored_count; + char *version = NULL; if (nreferenced <= 0) return; /* nothing to do */ @@ -96,12 +109,55 @@ recordMultipleDependencies(const ObjectAddress *depender, slot_init_count = 0; for (i = 0; i < nreferenced; i++, referenced++) { + bool ignore_systempin = false; + + if (track_version) + { + /* Only dependency on a collation is supported. */ + if (referenced->classId == CollationRelationId) + { + /* C and POSIX collations don't require tracking the version */ + if (referenced->objectId == C_COLLATION_OID || + referenced->objectId == POSIX_COLLATION_OID) + continue; + + /* + * We don't want to record redundant dependencies that are used + * to track versions to avoid redundant warnings in case of + * non-matching versions when those are checked. Note that + * callers have to take care of removing duplicated entries + * and calling CommandCounterIncrement() if the dependencies + * are registered in multiple calls. + * However, we may have already recorded a dependency on a + * collation that didn't required to track the version (if any + * of the *_pattern_ops opclasses was used). If we now see a + * dependency on the same collation that requires to track the + * version, we need to update it with the current version, + * which is done by dependencyExists(). + */ + version = get_collation_version_for_oid(referenced->objectId); + if (dependencyExists(depender, referenced, version)) + continue; + + /* + * Default collation is pinned, so we need to force recording + * the dependency to store the version. + */ + if (referenced->objectId == DEFAULT_COLLATION_OID) + ignore_systempin = true; + Assert(version); + } + } + else + Assert(!version); + /* * If the referenced object is pinned by the system, there's no real - * need to record dependencies on it. This saves lots of space in - * pg_depend, so it's worth the time taken to check. + * need to record dependencies on it, unless we need to record a + * version. This saves lots of space in pg_depend, so it's worth the + * time taken to check. */ - if (isObjectPinned(referenced, dependDesc)) + if (!ignore_systempin && isObjectPinned(referenced, dependDesc)) continue; if (slot_init_count < max_slots) @@ -493,7 +549,8 @@ changeDependencyFor(Oid classId, Oid objectId, } /* - * Adjust all dependency records to come from a different object of the same type + * Adjust all dependency records to come from a different object of the same + * type, optionally preserving the original referenced version. * * classId/oldObjectId specify the old referencing object. * newObjectId is the new referencing object (must be of class classId). @@ -501,8 +558,7 @@ changeDependencyFor(Oid classId, Oid objectId, * Returns the number of records updated. */ long -changeDependenciesOf(Oid classId, Oid oldObjectId, - Oid newObjectId) +changeDependenciesOf(Oid classId, Oid oldObjectId, Oid newObjectId) { long count = 0; Relation depRel; @@ -526,14 +582,47 @@ changeDependenciesOf(Oid classId, Oid oldObjectId, while (HeapTupleIsValid((tup = systable_getnext(scan)))) { - Form_pg_depend depform; + Form_pg_depend depform = (Form_pg_depend) GETSTRUCT(tup); + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + bool isnull = true; - /* make a modifiable copy */ - tup = heap_copytuple(tup); - depform = (Form_pg_depend) GETSTRUCT(tup); + memset(values, 0, sizeof(values)); + memset(nulls, 0, sizeof(nulls)); + memset(replaces, 0, sizeof(replaces)); - depform->objid = newObjectId; + values[Anum_pg_depend_objid - 1] = newObjectId; + replaces[Anum_pg_depend_objid - 1] = true; + + /* + * We assume that a version would exist for both the old and new + * object or none. + */ + heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + if (!isnull) + { + ObjectAddress depender, + referenced; + char *version; + + ObjectAddressSubSet(depender, depform->classid, + newObjectId, depform->objsubid); + ObjectAddressSubSet(referenced, depform->refclassid, + depform->refobjid, depform->refobjsubid); + + version = getDependencyVersion(&depender, &referenced); + if (version) + values[Anum_pg_depend_refobjversion - 1] = CStringGetTextDatum(version); + else + nulls[Anum_pg_depend_refobjversion - 1] = true; + replaces[Anum_pg_depend_refobjversion - 1] = true; + } + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); CatalogTupleUpdate(depRel, &tup->t_self, tup); heap_freetuple(tup); @@ -636,6 +725,151 @@ changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, return count; } +/* dependencyExists() + * + * Test if a record exists for the given depender and referenced addresses. + * + * If a record is found, also update the stored dependency version if it wasn't + * previously tracked. + */ +static bool +dependencyExists(const ObjectAddress *depender, + const ObjectAddress *referenced, + char *version) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + bool ret = false; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + char *cur_version = NULL; + Datum depversion; + bool isnull; + + ret = true; + + /* + * We only need the retrieve the current version if a new version + * was passed. + */ + if (version) + { + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + + cur_version = isnull ? NULL : TextDatumGetCString(depversion); + } + + /* Track version if it wasn't yet and we have a new version. */ + if (version && !cur_version) + { + Datum values[Natts_pg_depend]; + bool nulls[Natts_pg_depend]; + bool replaces[Natts_pg_depend]; + + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replaces, false, sizeof(replaces)); + + values[Anum_pg_depend_refobjversion - 1] = + CStringGetTextDatum(version); + replaces[Anum_pg_depend_refobjversion - 1] = true; + + tup = heap_modify_tuple(tup, RelationGetDescr(depRel), values, + nulls, replaces); + CatalogTupleUpdate(depRel, &tup->t_self, tup); + + heap_freetuple(tup); + } + else + { + Assert(strcmp(cur_version, version) == 0); + } + + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return ret; +} + +static char * +getDependencyVersion(const ObjectAddress *depender, + const ObjectAddress *referenced) +{ + Relation depRel; + ScanKeyData key[3]; + SysScanDesc scan; + HeapTuple tup; + Datum depversion; + char *version = NULL; + + ScanKeyInit(&key[0], + Anum_pg_depend_classid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->classId)); + ScanKeyInit(&key[1], + Anum_pg_depend_objid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(depender->objectId)); + ScanKeyInit(&key[2], + Anum_pg_depend_objsubid, + BTEqualStrategyNumber, F_INT4EQ, + Int32GetDatum(depender->objectSubId)); + + depRel = table_open(DependRelationId, RowExclusiveLock); + scan = systable_beginscan(depRel, DependDependerIndexId, true, + NULL, 3, key); + + while (HeapTupleIsValid(tup = systable_getnext(scan))) + { + Form_pg_depend foundDep = (Form_pg_depend) GETSTRUCT(tup); + bool isnull; + + if (foundDep->refclassid == referenced->classId && + foundDep->refobjid == referenced->objectId && + foundDep->refobjsubid == referenced->objectSubId) + { + depversion = heap_getattr(tup, Anum_pg_depend_refobjversion, + RelationGetDescr(depRel), &isnull); + version = isnull ? NULL : TextDatumGetCString(depversion); + break; + } + } + systable_endscan(scan); + table_close(depRel, RowExclusiveLock); + + return version; +} + /* * isObjectPinned() * diff --git a/src/backend/catalog/pg_type.c b/src/backend/catalog/pg_type.c index 0b04dff773..f2af6212e5 100644 --- a/src/backend/catalog/pg_type.c +++ b/src/backend/catalog/pg_type.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/relation.h" #include "access/table.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" @@ -512,6 +513,74 @@ TypeCreate(Oid newTypeOid, return address; } +/* Get a list of all distinct collations oid that the given type depends on. */ +List * +GetTypeCollations(Oid typeoid, bool non_deterministic_only) +{ + List *result = NIL; + HeapTuple tuple; + Form_pg_type typeTup; + + tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(typeoid)); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for type %u", typeoid); + typeTup = (Form_pg_type) GETSTRUCT(tuple); + + if (OidIsValid(typeTup->typcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(typeTup->typcollation)) + result = list_append_unique_oid(result, typeTup->typcollation); + } + else if (typeTup->typtype == TYPTYPE_COMPOSITE) + { + Relation rel = relation_open(typeTup->typrelid, AccessShareLock); + TupleDesc desc = RelationGetDescr(rel); + + for (int i = 0; i < RelationGetNumberOfAttributes(rel); i++) + { + Form_pg_attribute att = TupleDescAttr(desc, i); + + if (OidIsValid(att->attcollation)) + { + if (!non_deterministic_only || + !get_collation_isdeterministic(att->attcollation)) + result = list_append_unique_oid(result, att->attcollation); + } + else + result = list_concat_unique_oid(result, + GetTypeCollations(att->atttypid, + non_deterministic_only)); + } + + relation_close(rel, NoLock); + } + else if (typeTup->typtype == TYPTYPE_DOMAIN) + { + Assert(OidIsValid(typeTup->typbasetype)); + + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typbasetype, + non_deterministic_only)); + } + else if (typeTup->typtype == TYPTYPE_RANGE) + { + Oid rangeid = get_range_subtype(typeTup->oid); + + Assert(OidIsValid(rangeid)); + + result = list_concat_unique_oid(result, + GetTypeCollations(rangeid, non_deterministic_only)); + } + else if (OidIsValid(typeTup->typelem)) + result = list_concat_unique_oid(result, + GetTypeCollations(typeTup->typelem, non_deterministic_only)); + + ReleaseSysCache(tuple); + + return result; +} + /* * GenerateTypeDependencies: build the dependencies needed for a type * diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index 5ad8886e60..519f7a7df3 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -270,28 +270,12 @@ Datum pg_collation_actual_version(PG_FUNCTION_ARGS) { Oid collid = PG_GETARG_OID(0); - HeapTuple tp; - char *collcollate; - char collprovider; char *version; - tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(collid)); - if (!HeapTupleIsValid(tp)) - ereport(ERROR, - (errcode(ERRCODE_UNDEFINED_OBJECT), - errmsg("collation with OID %u does not exist", collid))); - - collcollate = pstrdup(NameStr(((Form_pg_collation) GETSTRUCT(tp))->collcollate)); - collprovider = ((Form_pg_collation) GETSTRUCT(tp))->collprovider; - - ReleaseSysCache(tp); - - version = get_collation_actual_version(collprovider, collcollate); + version = get_collation_version_for_oid(collid); + Assert(version); - if (version) - PG_RETURN_TEXT_P(cstring_to_text(version)); - else - PG_RETURN_NULL(); + PG_RETURN_TEXT_P(cstring_to_text(version)); } diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c index ddeec870d8..e1016609dc 100644 --- a/src/backend/commands/vacuum.c +++ b/src/backend/commands/vacuum.c @@ -31,6 +31,8 @@ #include "access/tableam.h" #include "access/transam.h" #include "access/xact.h" +#include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_database.h" #include "catalog/pg_inherits.h" @@ -632,6 +634,36 @@ vacuum_open_relation(Oid relid, RangeVar *relation, int options, rel_lock = false; } + /* + * Perform version sanity checks on the relation underlying indexes if + * it's not a VACUUM FULL, as VACUUM FULL will recreate the index and + * update the recorded collation version. + */ + if (!(options & VACOPT_FULL) && onerel && !IsSystemRelation(onerel) && + onerel->rd_rel->relhasindex) + { + List *indexoidlist; + ListCell *l; + + indexoidlist = RelationGetIndexList(onerel); + foreach(l, indexoidlist) + { + Oid indexoid = lfirst_oid(l); + Relation indexRelation; + + indexRelation = index_open(indexoid, AccessShareLock); + + /* Warn if any dependent collations' versions have moved. */ + if (!indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + + index_close(indexRelation, NoLock); + } + } + /* if relation is opened, leave */ if (onerel) return onerel; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index f9d0d67aa7..3a20dcdee5 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -28,6 +28,7 @@ #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/pg_am.h" #include "catalog/pg_proc.h" #include "catalog/pg_statistic_ext.h" @@ -198,6 +199,14 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, indexRelation = index_open(indexoid, lmode); index = indexRelation->rd_index; + /* Warn if any dependent collations' versions have moved. */ + if (!IsSystemRelation(relation) && + !indexRelation->rd_version_checked) + { + index_check_collation_versions(indexoid); + indexRelation->rd_version_checked = true; + } + /* * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 514e0fa0af..7e5fcb77df 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -57,7 +57,9 @@ #include "access/htup_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_control.h" +#include "catalog/pg_database.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/hsearch.h" @@ -139,6 +141,9 @@ static char *IsoLocaleName(const char *); /* MSVC specific */ static void icu_set_collation_attributes(UCollator *collator, const char *loc); #endif +static char *get_collation_actual_version(char collprovider, + const char *collcollate); + /* * pg_perm_setlocale * @@ -1630,7 +1635,7 @@ pg_newlocale_from_collation(Oid collid) * Get provider-specific collation version string for the given collation from * the operating system/library. */ -char * +static char * get_collation_actual_version(char collprovider, const char *collcollate) { char *collversion = NULL; @@ -1712,6 +1717,51 @@ get_collation_actual_version(char collprovider, const char *collcollate) return collversion; } +/* + * Get provider-specific collation version string given a collation OID. + * + * As this version is used in index dependency tracking, an empty string is + * returned when the version is unknown, to distinguish from dependencies where + * the version should not be tracked, represented by a NULL version. + */ +char * +get_collation_version_for_oid(Oid oid) +{ + HeapTuple tp; + char *version = NULL; + + Assert(oid != C_COLLATION_OID && oid != POSIX_COLLATION_OID); + + if (oid == DEFAULT_COLLATION_OID) + { + Form_pg_database dbform; + + tp = SearchSysCache1(DATABASEOID, ObjectIdGetDatum(MyDatabaseId)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for database %u", MyDatabaseId); + dbform = (Form_pg_database) GETSTRUCT(tp); + version = get_collation_actual_version(COLLPROVIDER_LIBC, + NameStr(dbform->datcollate)); + } + else + { + Form_pg_collation collform; + + tp = SearchSysCache1(COLLOID, ObjectIdGetDatum(oid)); + if (!HeapTupleIsValid(tp)) + elog(ERROR, "cache lookup failed for collation %u", oid); + collform = (Form_pg_collation) GETSTRUCT(tp); + version = get_collation_actual_version(collform->collprovider, + NameStr(collform->collcollate)); + } + + ReleaseSysCache(tp); + + if (!version) + return ""; + else + return version; +} #ifdef USE_ICU /* diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index 14d9eb2b5b..4ba8c8c637 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -13,6 +13,7 @@ #include "catalog/binary_upgrade.h" #include "catalog/heap.h" +#include "catalog/index.h" #include "catalog/namespace.h" #include "catalog/pg_type.h" #include "commands/extension.h" @@ -197,3 +198,27 @@ binary_upgrade_set_missing_value(PG_FUNCTION_ARGS) PG_RETURN_VOID(); } + +Datum +binary_upgrade_set_index_coll_version(PG_FUNCTION_ARGS) +{ + Oid relid; + Oid coll; + char *version; + + CHECK_IS_BINARY_UPGRADE; + + relid = PG_GETARG_OID(0); + + /* Detect if a collation is specified */ + if (PG_ARGISNULL(1)) + coll = InvalidOid; + else + coll = PG_GETARG_OID(1); + + version = TextDatumGetCString(PG_GETARG_TEXT_PP(2)); + + index_force_collation_versions(relid, coll, version); + + PG_RETURN_VOID(); +} diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 9061af81a3..153adc157d 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -42,6 +42,7 @@ #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" +#include "catalog/index.h" #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/partition.h" @@ -5934,6 +5935,7 @@ load_relcache_init_file(bool shared) rel->rd_idattr = NULL; rel->rd_pubactions = NULL; rel->rd_statvalid = false; + rel->rd_version_checked = false; rel->rd_statlist = NIL; rel->rd_fkeyvalid = false; rel->rd_fkeylist = NIL; diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile index 2532d9183a..c58ebaa681 100644 --- a/src/bin/pg_dump/Makefile +++ b/src/bin/pg_dump/Makefile @@ -16,6 +16,8 @@ subdir = src/bin/pg_dump top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu + override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS) LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport) diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 1017abbbe5..a94fbafec3 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -171,6 +171,7 @@ typedef struct _dumpOptions int sequence_data; /* dump sequence data even in schema-only mode */ int do_nothing; + int unknown_coll_compat; } DumpOptions; /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 2174d66c1f..cc5accc84c 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -46,6 +46,7 @@ #include "catalog/pg_attribute_d.h" #include "catalog/pg_cast_d.h" #include "catalog/pg_class_d.h" +#include "catalog/pg_collation_d.h" #include "catalog/pg_default_acl_d.h" #include "catalog/pg_largeobject_d.h" #include "catalog/pg_largeobject_metadata_d.h" @@ -285,6 +286,8 @@ static void binary_upgrade_extension_member(PQExpBuffer upgrade_buffer, static const char *getAttrName(int attrnum, TableInfo *tblInfo); static const char *fmtCopyColumnList(const TableInfo *ti, PQExpBuffer buffer); static bool nonemptyReloptions(const char *reloptions); +static void appendIndexCollationVersion(PQExpBuffer buffer, IndxInfo *indxinfo, + int enc, int unknown_coll_compat); static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions, const char *prefix, Archive *fout); static char *get_synchronized_snapshot(Archive *fout); @@ -386,6 +389,7 @@ main(int argc, char **argv) {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, {"include-foreign-data", required_argument, NULL, 11}, + {"unknown-collations-binary-compatible", no_argument, &dopt.unknown_coll_compat, 1}, {NULL, 0, NULL, 0} }; @@ -713,6 +717,10 @@ main(int argc, char **argv) if (archiveFormat != archDirectory && numWorkers > 1) fatal("parallel backup only supported by the directory format"); + /* Unknown collation versions can only be ignored in binary upgrade mode */ + if (dopt.unknown_coll_compat && !dopt.binary_upgrade) + fatal("option --unknown-collations-binary-compatible only works in binary upgrade mode"); + /* Open the output file */ fout = CreateArchive(filename, archiveFormat, compressLevel, dosync, archiveMode, setupDumpWorker); @@ -7024,7 +7032,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) i_tablespace, i_indreloptions, i_indstatcols, - i_indstatvals; + i_indstatvals, + i_inddependoids, + i_inddependversions; int ntups; for (i = 0; i < numTables; i++) @@ -7060,7 +7070,62 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) * is not. */ resetPQExpBuffer(query); - if (fout->remoteVersion >= 110000) + if (fout->remoteVersion >= 140000) + { + appendPQExpBuffer(query, + "SELECT t.tableoid, t.oid, " + "t.relname AS indexname, " + "inh.inhparent AS parentidx, " + "pg_catalog.pg_get_indexdef(i.indexrelid) AS indexdef, " + "i.indnkeyatts AS indnkeyatts, " + "i.indnatts AS indnatts, " + "i.indkey, i.indisclustered, " + "i.indisreplident, " + "c.contype, c.conname, " + "c.condeferrable, c.condeferred, " + "c.tableoid AS contableoid, " + "c.oid AS conoid, " + "pg_catalog.pg_get_constraintdef(c.oid, false) AS condef, " + "(SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, " + "t.reloptions AS indreloptions, " + "(SELECT pg_catalog.array_agg(attnum ORDER BY attnum) " + " FROM pg_catalog.pg_attribute " + " WHERE attrelid = i.indexrelid AND " + " attstattarget >= 0) AS indstatcols," + "(SELECT pg_catalog.array_agg(attstattarget ORDER BY attnum) " + " FROM pg_catalog.pg_attribute " + " WHERE attrelid = i.indexrelid AND " + " attstattarget >= 0) AS indstatvals, " + "(SELECT pg_catalog.array_agg(refobjid ORDER BY refobjid) " + " FROM pg_catalog.pg_depend " + " WHERE classid = " CppAsString2(RelationRelationId) " AND " + " objid = i.indexrelid AND " + " objsubid = 0 AND " + " refclassid = " CppAsString2(CollationRelationId) " AND " + " refobjversion IS NOT NULL) AS inddependoids, " + "(SELECT pg_catalog.array_agg(quote_literal(refobjversion) ORDER BY refobjid) " + " FROM pg_catalog.pg_depend " + " WHERE classid = " CppAsString2(RelationRelationId) " AND " + " objid = i.indexrelid AND " + " objsubid = 0 AND " + " refclassid = " CppAsString2(CollationRelationId) " AND " + " refobjversion IS NOT NULL) AS inddependversions " + "FROM pg_catalog.pg_index i " + "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " + "JOIN pg_catalog.pg_class t2 ON (t2.oid = i.indrelid) " + "LEFT JOIN pg_catalog.pg_constraint c " + "ON (i.indrelid = c.conrelid AND " + "i.indexrelid = c.conindid AND " + "c.contype IN ('p','u','x')) " + "LEFT JOIN pg_catalog.pg_inherits inh " + "ON (inh.inhrelid = indexrelid) " + "WHERE i.indrelid = '%u'::pg_catalog.oid " + "AND (i.indisvalid OR t2.relkind = 'p') " + "AND i.indisready " + "ORDER BY indexname", + tbinfo->dobj.catId.oid); + } + else if (fout->remoteVersion >= 110000) { appendPQExpBuffer(query, "SELECT t.tableoid, t.oid, " @@ -7085,7 +7150,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) "(SELECT pg_catalog.array_agg(attstattarget ORDER BY attnum) " " FROM pg_catalog.pg_attribute " " WHERE attrelid = i.indexrelid AND " - " attstattarget >= 0) AS indstatvals " + " attstattarget >= 0) AS indstatvals, " + "' ' AS inddependoids, " + "' ' AS inddependversions " "FROM pg_catalog.pg_index i " "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " "JOIN pg_catalog.pg_class t2 ON (t2.oid = i.indrelid) " @@ -7124,7 +7191,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) "(SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, " "t.reloptions AS indreloptions, " "'' AS indstatcols, " - "'' AS indstatvals " + "'' AS indstatvals, " + "' ' AS inddependoids, " + "' ' AS inddependversions " "FROM pg_catalog.pg_index i " "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " "LEFT JOIN pg_catalog.pg_constraint c " @@ -7159,7 +7228,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) "(SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, " "t.reloptions AS indreloptions, " "'' AS indstatcols, " - "'' AS indstatvals " + "'' AS indstatvals, " + "' ' AS inddependoids, " + "' ' AS inddependversions " "FROM pg_catalog.pg_index i " "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " "LEFT JOIN pg_catalog.pg_constraint c " @@ -7190,7 +7261,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) "(SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, " "t.reloptions AS indreloptions, " "'' AS indstatcols, " - "'' AS indstatvals " + "'' AS indstatvals, " + "' ' AS inddependoids, " + "' ' AS inddependversions " "FROM pg_catalog.pg_index i " "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " "LEFT JOIN pg_catalog.pg_depend d " @@ -7224,7 +7297,9 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) "(SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, " "null AS indreloptions, " "'' AS indstatcols, " - "'' AS indstatvals " + "'' AS indstatvals, " + "' ' AS inddependoids, " + "' ' AS inddependversions " "FROM pg_catalog.pg_index i " "JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) " "LEFT JOIN pg_catalog.pg_depend d " @@ -7264,6 +7339,8 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) i_indreloptions = PQfnumber(res, "indreloptions"); i_indstatcols = PQfnumber(res, "indstatcols"); i_indstatvals = PQfnumber(res, "indstatvals"); + i_inddependoids = PQfnumber(res, "inddependoids"); + i_inddependversions = PQfnumber(res, "inddependversions"); tbinfo->indexes = indxinfo = (IndxInfo *) pg_malloc(ntups * sizeof(IndxInfo)); @@ -7289,6 +7366,8 @@ getIndexes(Archive *fout, TableInfo tblinfo[], int numTables) indxinfo[j].indreloptions = pg_strdup(PQgetvalue(res, j, i_indreloptions)); indxinfo[j].indstatcols = pg_strdup(PQgetvalue(res, j, i_indstatcols)); indxinfo[j].indstatvals = pg_strdup(PQgetvalue(res, j, i_indstatvals)); + indxinfo[j].inddependoids = pg_strdup(PQgetvalue(res, j, i_inddependoids)); + indxinfo[j].inddependversions = pg_strdup(PQgetvalue(res, j, i_inddependversions)); indxinfo[j].indkeys = (Oid *) pg_malloc(indxinfo[j].indnattrs * sizeof(Oid)); parseOidArray(PQgetvalue(res, j, i_indkey), indxinfo[j].indkeys, indxinfo[j].indnattrs); @@ -16353,7 +16432,8 @@ dumpIndex(Archive *fout, IndxInfo *indxinfo) /* * If there's an associated constraint, don't dump the index per se, but - * do dump any comment for it. (This is safe because dependency ordering + * do dump any comment, or in binary upgrade mode dependency on a + * collation version for it. (This is safe because dependency ordering * will have ensured the constraint is emitted first.) Note that the * emitted comment has to be shown as depending on the constraint, not the * index, in such cases. @@ -16420,6 +16500,10 @@ dumpIndex(Archive *fout, IndxInfo *indxinfo) "pg_catalog.pg_class", "INDEX", qqindxname); + if (dopt->binary_upgrade) + appendIndexCollationVersion(q, indxinfo, fout->encoding, + dopt->unknown_coll_compat); + /* If the index defines identity, we need to record that. */ if (indxinfo->indisreplident) { @@ -16448,6 +16532,21 @@ dumpIndex(Archive *fout, IndxInfo *indxinfo) if (indstatvalsarray) free(indstatvalsarray); } + else if (dopt->binary_upgrade) + { + appendIndexCollationVersion(q, indxinfo, fout->encoding, + dopt->unknown_coll_compat); + + if (indxinfo->dobj.dump & DUMP_COMPONENT_DEFINITION) + ArchiveEntry(fout, indxinfo->dobj.catId, indxinfo->dobj.dumpId, + ARCHIVE_OPTS(.tag = indxinfo->dobj.name, + .namespace = tbinfo->dobj.namespace->dobj.name, + .tablespace = indxinfo->tablespace, + .owner = tbinfo->rolname, + .description = "INDEX", + .section = SECTION_POST_DATA, + .createStmt = q->data)); + } /* Dump Index Comments */ if (indxinfo->dobj.dump & DUMP_COMPONENT_COMMENT) @@ -18434,6 +18533,77 @@ nonemptyReloptions(const char *reloptions) return (reloptions != NULL && strlen(reloptions) > 2); } +/* + * Format inddependoids and inddependversions arrays and append it to the given + * buffer in the form of binary_upgrade_set_index_coll_version() calls. + */ +static void +appendIndexCollationVersion(PQExpBuffer buffer, IndxInfo *indxinfo, int enc, + int unknown_coll_compat) +{ + char *inddependoids = indxinfo->inddependoids; + char *inddependversions = indxinfo->inddependversions; + char **inddependoidsarray = NULL; + char **inddependversionsarray = NULL; + int ninddependoids; + int ninddependversions; + int i; + + /* + * for older versions that don't record the collation depndency, issue a + * statement to mark the collation version as unknown + */ + if (strcmp(inddependoids, " ") == 0) + { + /* + * do not issue UNKNOWN VERSION if caller specified that those are + * compatible + */ + if (unknown_coll_compat) + return; + + Assert(strcmp(inddependversions, " ") == 0); + + appendPQExpBufferStr(buffer, "\n-- For binary upgrade, restore dependent collation version.\n"); + appendPQExpBuffer(buffer, "SELECT " + "pg_catalog.binary_upgrade_set_index_coll_version(%d, NULL, '');", + indxinfo->dobj.catId.oid); + return; + } + + parsePGArray(inddependoids, &inddependoidsarray, &ninddependoids); + parsePGArray(inddependversions, &inddependversionsarray, &ninddependversions); + + Assert(ninddependoids == ninddependversions); + + for (i = 0; i < ninddependoids; i++) + { + /* + * If there was an unknown version dependency recorded for this + * collation and the caller asked to mark those as depending on the + * current version, don't emit a binary_upgrade_set_index_coll_version + * function call. + */ + if ((strcmp(inddependversionsarray[i], "''")) == 0 + && unknown_coll_compat) + { + continue; + } + + appendPQExpBufferStr(buffer, "\n-- For binary upgrade, restore dependent collation version.\n"); + appendPQExpBuffer(buffer, "SELECT " + "pg_catalog.binary_upgrade_set_index_coll_version(%d, %s, %s);", + indxinfo->dobj.catId.oid, + inddependoidsarray[i], + inddependversionsarray[i]); + } + + if (inddependoidsarray) + free(inddependoidsarray); + if (inddependversionsarray) + free(inddependversionsarray); +} + /* * Format a reloptions array and append it to the given buffer. * diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index e0b42e8391..623814d1c5 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -366,6 +366,9 @@ typedef struct _indxInfo int indnattrs; /* total number of index attributes */ Oid *indkeys; /* In spite of the name 'indkeys' this field * contains both key and nonkey attributes */ + char *inddependoids; /* oids of collation this index depends on */ + char *inddependversions; /* version of collation this index depends + * on */ bool indisclustered; bool indisreplident; Oid parentidx; /* if a partition, parent index OID */ diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index ec63662060..5f323efb1f 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -53,6 +53,24 @@ my %pgdump_runs = ( "$tempdir/binary_upgrade.dump", ], }, + binary_coll_compatible => { + dump_cmd => [ + 'pg_dump', + '--no-sync', + '--format=custom', + "--file=$tempdir/binary_coll_compatible.dump", + '-w', + '--schema-only', + '--binary-upgrade', + '--unknown-collations-binary-compatible', + '-d', 'postgres', # alternative way to specify database + ], + restore_cmd => [ + 'pg_restore', '-Fc', '--verbose', + "--file=$tempdir/binary_coll_compatible.sql", + "$tempdir/binary_coll_compatible.dump", + ], + }, clean => { dump_cmd => [ 'pg_dump', @@ -387,6 +405,7 @@ my %dump_test_schema_runs = ( # are flags used to exclude specific items (ACLs, blobs, etc). my %full_runs = ( binary_upgrade => 1, + binary_coll_compatible => 1, clean => 1, clean_if_exists => 1, createdb => 1, @@ -920,9 +939,10 @@ my %tests = ( test_schema_plus_blobs => 1, }, unlike => { - binary_upgrade => 1, - no_blobs => 1, - schema_only => 1, + binary_upgrade => 1, + binary_coll_compatible => 1, + no_blobs => 1, + schema_only => 1, }, }, @@ -1184,6 +1204,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, exclude_test_table => 1, exclude_test_table_data => 1, @@ -1209,6 +1230,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -1244,6 +1266,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -1266,6 +1289,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -1287,6 +1311,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -1308,6 +1333,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -1674,6 +1700,7 @@ my %tests = ( { %full_runs, %dump_test_schema_runs, section_pre_data => 1, }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, }, }, @@ -1688,7 +1715,7 @@ my %tests = ( \n.*^ \QALTER TYPE dump_test.planets ADD VALUE 'mars';\E \n/xms, - like => { binary_upgrade => 1, }, + like => { binary_upgrade => 1, binary_coll_compatible => 1, }, }, 'CREATE TYPE dump_test.textrange AS RANGE' => { @@ -2356,6 +2383,7 @@ my %tests = ( { %full_runs, %dump_test_schema_runs, section_pre_data => 1, }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, }, }, @@ -2549,6 +2577,7 @@ my %tests = ( }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, }, }, @@ -2617,6 +2646,7 @@ my %tests = ( /xm, like => { binary_upgrade => 1, + binary_coll_compatible => 1, clean => 1, clean_if_exists => 1, createdb => 1, @@ -2688,6 +2718,7 @@ my %tests = ( /xm, like => { binary_upgrade => 1, + binary_coll_compatible => 1, clean => 1, clean_if_exists => 1, createdb => 1, @@ -3155,6 +3186,7 @@ my %tests = ( { %full_runs, %dump_test_schema_runs, section_post_data => 1, }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -3170,6 +3202,7 @@ my %tests = ( { %full_runs, %dump_test_schema_runs, section_post_data => 1, }, unlike => { binary_upgrade => 1, + binary_coll_compatible => 1, exclude_dump_test_schema => 1, schema_only => 1, }, @@ -3302,16 +3335,53 @@ my %tests = ( %full_runs, %dump_test_schema_runs, section_pre_data => 1, }, unlike => { exclude_dump_test_schema => 1 }, + }, + + "binary_upgrade_set_index_coll_version(oid, oid, 'not_a_version')" => { + create_order => 101, + create_sql => ' + CREATE TABLE dump_test.regress_table_coll(id integer, val text); + CREATE INDEX regress_coll_idx1 ON dump_test.regress_table_coll(val COLLATE "fr-x-icu"); + UPDATE pg_depend SET refobjversion = \'not_a_version\' WHERE refobjversion IS NOT NULL AND objid::regclass::text = \'dump_test.regress_coll_idx1\';', + regexp => qr/^ + \QCREATE INDEX regress_coll_idx1 ON dump_test.regress_table_coll USING btree (val COLLATE "fr-x-icu");\E\n + \n + \Q-- For binary upgrade, restore dependent collation version.\E\n + \QSELECT pg_catalog.binary_upgrade_set_index_coll_version\E \(\d+,\ \d+,\ 'not_a_version'\);/xm, + like => { binary_upgrade => 1, binary_coll_compatible => 1, }, + icu => 1, + }, + "binary_upgrade_set_index_coll_version(?, ?, '')" => { + create_order => 102, + create_sql => ' + CREATE TABLE dump_test.regress_table_coll_no_ver(id integer, val text); + CREATE INDEX regress_coll_no_ver_idx1 ON dump_test.regress_table_coll_no_ver(val COLLATE "fr-x-icu"); + UPDATE pg_depend SET refobjversion = \'\' WHERE refobjversion IS NOT NULL AND objid::regclass::text = \'dump_test.regress_coll_no_ver_idx1\';', + regexp => qr/SELECT pg_catalog.binary_upgrade_set_index_coll_version\(\d+, \d+, ''\)/, + like => { binary_upgrade => 1}, + # should not appear in binary_coll_compatible case! + unlike => { binary_coll_compatible => 1}, + icu => 1, }); ######################################### # Create a PG instance to test actually dumping from -my $node = get_new_node('main'); -$node->init; -$node->start; +my $main_node = get_new_node('main'); +$main_node->init; +$main_node->start; + +my $port = $main_node->port; + +# And another instance to validate the binary dump +my $bin_node = get_new_node('binary'); +$bin_node->init; +$bin_node->start; -my $port = $node->port; +my $bin_port = $bin_node->port; + +# and add a $node variable pointing to main_node for now +my $node = $main_node; # We need to see if this system supports CREATE COLLATION or not # If it doesn't then we will skip all the COLLATION-related tests. @@ -3335,6 +3405,10 @@ $node->psql('postgres', 'create database regress_pg_dump_test;'); # command_fails_like is actually 2 tests) my $num_tests = 12; +# 4 more tests for restoring globals and binary_upgrade dump, dumping it again +# and regenerating the sql file +$num_tests+= 4; + foreach my $run (sort keys %pgdump_runs) { my $test_key = $run; @@ -3385,16 +3459,29 @@ foreach my $run (sort keys %pgdump_runs) next; } + # Skip any icu-related commands if there is no icu support + if ($ENV{with_icu} ne 'yes' && defined($tests{$test}->{icu})) + { + next; + } + # If there is a like entry, but no unlike entry, then we will test the like case if ($tests{$test}->{like}->{$test_key} && !defined($tests{$test}->{unlike}->{$test_key})) { $num_tests++; + + # binary_upgrade tests are also run after being restored and + # re-dumped. + $num_tests++ if ($test_key eq 'binary_upgrade'); } else { # We will test everything that isn't a 'like' $num_tests++; + # binary_upgrade tests are also run after being restored and + # re-dumped. + $num_tests++ if ($test_key eq 'binary_upgrade'); } } } @@ -3442,6 +3529,12 @@ foreach my $test ( next; } + # Skip any icu-related commands if there is no icu support + if ($ENV{with_icu} ne 'yes' && defined($tests{$test}->{icu})) + { + next; + } + # Add terminating semicolon $create_sql{$test_db} .= $tests{$test}->{create_sql} . ";"; } @@ -3495,79 +3588,116 @@ command_fails_like( ######################################### # Run all runs -foreach my $run (sort keys %pgdump_runs) +foreach my $pass (1, 2) { - my $test_key = $run; - my $run_db = 'postgres'; - - $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, - "$run: pg_dump runs"); - - if ($pgdump_runs{$run}->{restore_cmd}) + foreach my $run (sort keys %pgdump_runs) { - $node->command_ok(\@{ $pgdump_runs{$run}->{restore_cmd} }, - "$run: pg_restore runs"); - } - - if ($pgdump_runs{$run}->{test_key}) - { - $test_key = $pgdump_runs{$run}->{test_key}; - } - - my $output_file = slurp_file("$tempdir/${run}.sql"); + my $test_key = $run; + my $run_db = 'postgres'; - ######################################### - # Run all tests where this run is included - # as either a 'like' or 'unlike' test. + # we only test binary upgrade on the 2nd pass + next if ($pass == 2 and $test_key ne 'binary_upgrade'); - foreach my $test (sort keys %tests) - { - my $test_db = 'postgres'; + $node->command_ok(\@{ $pgdump_runs{$run}->{dump_cmd} }, + "$run: pg_dump runs"); - if (defined($pgdump_runs{$run}->{database})) + if ($pgdump_runs{$run}->{restore_cmd}) { - $run_db = $pgdump_runs{$run}->{database}; + $node->command_ok(\@{ $pgdump_runs{$run}->{restore_cmd} }, + "$run: pg_restore runs"); } - if (defined($tests{$test}->{database})) + if ($pgdump_runs{$run}->{test_key}) { - $test_db = $tests{$test}->{database}; + $test_key = $pgdump_runs{$run}->{test_key}; } - # Skip any collation-related commands if there is no collation support - if (!$collation_support && defined($tests{$test}->{collation})) - { - next; - } + my $output_file = slurp_file("$tempdir/${run}.sql"); - if ($run_db ne $test_db) - { - next; - } + ######################################### + # Run all tests where this run is included + # as either a 'like' or 'unlike' test. - # Run the test listed as a like, unless it is specifically noted - # as an unlike (generally due to an explicit exclusion or similar). - if ($tests{$test}->{like}->{$test_key} - && !defined($tests{$test}->{unlike}->{$test_key})) + foreach my $test (sort keys %tests) { - if (!ok($output_file =~ $tests{$test}->{regexp}, - "$run: should dump $test")) + my $test_db = 'postgres'; + + if (defined($pgdump_runs{$run}->{database})) { - diag("Review $run results in $tempdir"); + $run_db = $pgdump_runs{$run}->{database}; } - } - else - { - if (!ok($output_file !~ $tests{$test}->{regexp}, - "$run: should not dump $test")) + + if (defined($tests{$test}->{database})) + { + $test_db = $tests{$test}->{database}; + } + + # Skip any collation-related commands if there is no collation support + if (!$collation_support && defined($tests{$test}->{collation})) + { + next; + } + + # Skip any icu-related commands if there is no icu support + if ($ENV{with_icu} ne 'yes' && defined($tests{$test}->{icu})) + { + next; + } + + if ($run_db ne $test_db) + { + next; + } + + # Run the test listed as a like, unless it is specifically noted + # as an unlike (generally due to an explicit exclusion or similar). + if ($tests{$test}->{like}->{$test_key} + && !defined($tests{$test}->{unlike}->{$test_key})) { - diag("Review $run results in $tempdir"); + if (!ok($output_file =~ $tests{$test}->{regexp}, + "$run: should dump $test")) + { + diag("Review $run results in $tempdir"); + } + } + else + { + if (!ok($output_file !~ $tests{$test}->{regexp}, + "$run: should not dump $test")) + { + diag("Review $run results in $tempdir"); + } } } } + + # After all dump have been generated, restore the binary_upgrade dump with + # the required global objects on a suitable node, and continue with the 2nd + # pass. + if ($pass == 1) + { + # Stop the original database instance as we don't need it anymore. + $node->stop('fast'); + + $bin_node->command_ok(\@{['psql', + "-d", "postgres", "-f", "$tempdir/pg_dumpall_globals.sql"]}, + "Restore globals"); + + $bin_node->stop('fast'); + $bin_node->start(binary_start => 1); + $bin_node->command_ok(\@{['pg_restore', '-p', $bin_port, + '-d', 'postgres', + "$tempdir/binary_upgrade.dump"]}, + "Restore the binary_upgrade dump"); + $bin_node->stop('fast'); + $bin_node->start; + + # And change $node to point to the freshly restored node. + $node = $bin_node; + } } ######################################### # Stop the database instance, which will be removed at the end of the tests. -$node->stop('fast'); +$bin_node->stop('fast'); diff --git a/src/bin/pg_upgrade/dump.c b/src/bin/pg_upgrade/dump.c index 4d730adfe2..672ecda169 100644 --- a/src/bin/pg_upgrade/dump.c +++ b/src/bin/pg_upgrade/dump.c @@ -52,9 +52,11 @@ generate_old_dump(void) parallel_exec_prog(log_file_name, NULL, "\"%s/pg_dump\" %s --schema-only --quote-all-identifiers " - "--binary-upgrade --format=custom %s --file=\"%s\" %s", + "--binary-upgrade --format=custom %s %s --file=\"%s\" %s", new_cluster.bindir, cluster_conn_opts(&old_cluster), log_opts.verbose ? "--verbose" : "", + user_opts.coll_compat ? + "--unknown-collations-binary-compatible" : "", sql_file_name, escaped_connstr.data); termPQExpBuffer(&escaped_connstr); diff --git a/src/bin/pg_upgrade/option.c b/src/bin/pg_upgrade/option.c index aca1ee8b48..c7e291f7e7 100644 --- a/src/bin/pg_upgrade/option.c +++ b/src/bin/pg_upgrade/option.c @@ -56,6 +56,7 @@ parseCommandLine(int argc, char *argv[]) {"socketdir", required_argument, NULL, 's'}, {"verbose", no_argument, NULL, 'v'}, {"clone", no_argument, NULL, 1}, + {"collation-binary-compatible", no_argument, NULL, 2}, {NULL, 0, NULL, 0} }; @@ -203,6 +204,10 @@ parseCommandLine(int argc, char *argv[]) user_opts.transfer_mode = TRANSFER_MODE_CLONE; break; + case 2: + user_opts.coll_compat = true; + break; + default: fprintf(stderr, _("Try \"%s --help\" for more information.\n"), os_info.progname); @@ -307,6 +312,8 @@ usage(void) printf(_(" -v, --verbose enable verbose internal logging\n")); printf(_(" -V, --version display version information, then exit\n")); printf(_(" --clone clone instead of copying files to new cluster\n")); + printf(_(" --collation-binary-compatible mark collations as depending on current collation\n" + " versions rather than unknown if they're unknown\n")); printf(_(" -?, --help show this help, then exit\n")); printf(_("\n" "Before running pg_upgrade you must:\n" diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 8b90cefbe0..68e637ce8a 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -292,6 +292,8 @@ typedef struct transferMode transfer_mode; /* copy files or link them? */ int jobs; /* number of processes/threads to use */ char *socketdir; /* directory to use for Unix sockets */ + bool coll_compat; /* should we skip marking index collations as + * unknown version */ } UserOpts; typedef struct diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h index 3baa5e498a..2966339498 100644 --- a/src/include/catalog/dependency.h +++ b/src/include/catalog/dependency.h @@ -130,6 +130,13 @@ typedef enum ObjectClass #define LAST_OCLASS OCLASS_TRANSFORM +/* Struct describing one forced collation version dependency */ +typedef struct NewCollationVersionDependency +{ + char *version; /* forced collation version */ + Oid oid; /* target collation oid */ +} NewCollationVersionDependency; + /* flag bits for performDeletion/performMultipleDeletions: */ #define PERFORM_DELETION_INTERNAL 0x0001 /* internal action */ #define PERFORM_DELETION_CONCURRENTLY 0x0002 /* concurrent drop */ @@ -160,7 +167,8 @@ extern void recordDependencyOnSingleRelExpr(const ObjectAddress *depender, Node *expr, Oid relId, DependencyType behavior, DependencyType self_behavior, - bool reverse_self); + bool reverse_self, + bool track_version); extern ObjectClass getObjectClass(const ObjectAddress *object); @@ -180,17 +188,29 @@ extern void sort_object_addresses(ObjectAddresses *addrs); extern void free_object_addresses(ObjectAddresses *addrs); +typedef char *(*VisitDependentObjectsFun) (const ObjectAddress *otherObject, + const char *version, + void *userdata); + +extern void visitDependentObjects(const ObjectAddress *object, + VisitDependentObjectsFun callback, + void *userdata); + /* in pg_depend.c */ extern void recordDependencyOn(const ObjectAddress *depender, const ObjectAddress *referenced, DependencyType behavior); +extern void recordDependencyOnCollations(ObjectAddress *myself, + List *collations, + bool record_version); + extern void recordMultipleDependencies(const ObjectAddress *depender, const ObjectAddress *referenced, int nreferenced, - const char *version, - DependencyType behavior); + DependencyType behavior, + bool track_version); extern void recordDependencyOnCurrentExtension(const ObjectAddress *object, bool isReplace); @@ -209,10 +229,9 @@ extern long changeDependencyFor(Oid classId, Oid objectId, Oid refClassId, Oid oldRefObjectId, Oid newRefObjectId); -extern long changeDependenciesOf(Oid classId, Oid oldObjectId, +long changeDependenciesOf(Oid classId, Oid oldObjectId, Oid newObjectId); - -extern long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, +long changeDependenciesOn(Oid refClassId, Oid oldRefObjectId, Oid newRefObjectId); extern Oid getExtensionOfObject(Oid classId, Oid objectId); diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index f58e8675f3..9b4de26514 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -121,6 +121,11 @@ extern void FormIndexDatum(IndexInfo *indexInfo, Datum *values, bool *isnull); +extern void index_check_collation_versions(Oid relid); + +extern void index_force_collation_versions(Oid indexid, Oid coll, + char *version); + extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, @@ -131,6 +136,8 @@ extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); extern void index_set_state_flags(Oid indexId, IndexStateFlagsAction action); +extern void index_update_collation_versions(Oid relid); + extern Oid IndexGetRelation(Oid indexId, bool missing_ok); extern void reindex_index(Oid indexId, bool skip_constraint_checks, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f48f5fb4d9..daf29aa6c5 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -10367,6 +10367,10 @@ proname => 'binary_upgrade_set_missing_value', provolatile => 'v', proparallel => 'u', prorettype => 'void', proargtypes => 'oid text text', prosrc => 'binary_upgrade_set_missing_value' }, +{ oid => '8178', descr => 'for use by pg_upgrade', + proname => 'binary_upgrade_set_index_coll_version', provolatile => 'v', + proparallel => 'u', prorettype => 'void', proargtypes => 'oid oid text', + prosrc => 'binary_upgrade_set_index_coll_version' }, # conversion functions { oid => '4302', diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h index 7b37562648..2d511c5cba 100644 --- a/src/include/catalog/pg_type.h +++ b/src/include/catalog/pg_type.h @@ -361,6 +361,8 @@ extern void GenerateTypeDependencies(HeapTuple typeTuple, bool isDependentType, bool rebuild); +extern List *GetTypeCollations(Oid typeObjectid, bool non_deterministic_only); + extern void RenameTypeInternal(Oid typeOid, const char *newTypeName, Oid typeNamespace); diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index 9cb7d91ddf..96da132c03 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -103,7 +103,7 @@ typedef struct pg_locale_struct *pg_locale_t; extern pg_locale_t pg_newlocale_from_collation(Oid collid); -extern char *get_collation_actual_version(char collprovider, const char *collcollate); +extern char *get_collation_version_for_oid(Oid collid); #ifdef USE_ICU extern int32_t icu_to_uchar(UChar **buff_uchar, const char *buff, size_t nbytes); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 0b5957ba02..f79667d651 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -63,6 +63,7 @@ typedef struct RelationData bool rd_indexvalid; /* is rd_indexlist valid? (also rd_pkindex and * rd_replidindex) */ bool rd_statvalid; /* is rd_statlist valid? */ + bool rd_version_checked; /* has version check being done yet? */ /*---------- * rd_createSubid is the ID of the highest subtransaction the rel has diff --git a/src/test/Makefile b/src/test/Makefile index efb206aa75..4ef0d11702 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -12,7 +12,8 @@ subdir = src/test top_builddir = ../.. include $(top_builddir)/src/Makefile.global -SUBDIRS = perl regress isolation modules authentication recovery subscription +SUBDIRS = perl regress isolation modules authentication recovery subscription \ + locale # Test suites that are not safe by default but can be run if selected # by the user via the whitespace-separated list in variable @@ -37,7 +38,7 @@ endif # clean" etc to recurse into them. (We must filter out those that we # have conditionally included into SUBDIRS above, else there will be # make confusion.) -ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl) +ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap thread ssl) # We want to recurse to all subdirs for all standard targets, except that # installcheck and install should not recurse into the subdirectory "modules". diff --git a/src/test/locale/.gitignore b/src/test/locale/.gitignore index 620d3df425..64e1bf2a80 100644 --- a/src/test/locale/.gitignore +++ b/src/test/locale/.gitignore @@ -1 +1,2 @@ /test-ctype +/tmp_check/ diff --git a/src/test/locale/Makefile b/src/test/locale/Makefile index 22a45b65f2..73495cf16b 100644 --- a/src/test/locale/Makefile +++ b/src/test/locale/Makefile @@ -4,6 +4,7 @@ subdir = src/test/locale top_builddir = ../../.. include $(top_builddir)/src/Makefile.global +export with_icu PROGS = test-ctype DIRS = de_DE.ISO8859-1 gr_GR.ISO8859-7 koi8-r koi8-to-win1251 @@ -19,3 +20,9 @@ clean distclean maintainer-clean: # These behave like installcheck targets. check-%: all @$(MAKE) -C `echo $@ | sed 's/^check-//'` test + +check: + $(prove_check) + +installcheck: + $(prove_installcheck) diff --git a/src/test/locale/t/001_index.pl b/src/test/locale/t/001_index.pl new file mode 100644 index 0000000000..468fbb63b6 --- /dev/null +++ b/src/test/locale/t/001_index.pl @@ -0,0 +1,76 @@ +use strict; +use warnings; + +use Config; +use PostgresNode; +use TestLib; +use Test::More; + +if ($ENV{with_icu} eq 'yes') +{ + plan tests => 12; +} +else +{ + plan skip_all => 'ICU not supported by this build'; +} + +#### Set up the server + +note "setting up data directory"; +my $node = get_new_node('main'); +$node->init; + +$ENV{PGHOST} = $node->host; +$ENV{PGPORT} = $node->port; +$node->start; + +sub test_index +{ + my ($err_like, $err_comm) = @_; + + my ($ret, $out, $err) = $node->psql('postgres', + 'SET enable_seqscan = 0;' + . "EXPLAIN SELECT val FROM icu1 WHERE val = '0'"); + + is($ret, 0, 'EXPLAIN should succeed.'); + like($out, qr/icu1_fr/, 'Index icu1_fr should be used.'); + like($err, $err_like, $err_comm); +} + +$node->safe_psql('postgres', + 'CREATE TABLE icu1(val text);' + . 'INSERT INTO icu1 SELECT i::text FROM generate_series(1, 10000) i;' + . 'CREATE INDEX icu1_fr ON icu1 (val COLLATE "fr-x-icu");'); +$node->safe_psql('postgres', 'VACUUM ANALYZE icu1;'); + +test_index(qr/^$/, 'No warning should be raised'); + +# Simulate different collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = 'not_a_version'" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" version "not_a_version", but the current version is/, + 'Different collation version warning should be raised.'); + +# Simulate unknown collation version +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = ''" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +# Simulate previously unhandled collation versioning +$node->safe_psql('postgres', + "UPDATE pg_depend SET refobjversion = NULL" + . " WHERE refobjversion IS NOT NULL" + . " AND objid::regclass::text = 'icu1_fr';"); + +test_index(qr/index "icu1_fr" depends on collation "fr-x-icu" with an unknown version, and the current version is/, + 'Unknown collation version warning should be raised.'); + +$node->stop; diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm index 1488bffa2b..8c54e12268 100644 --- a/src/test/perl/PostgresNode.pm +++ b/src/test/perl/PostgresNode.pm @@ -765,10 +765,14 @@ sub start local %ENV = %ENV; delete $ENV{PGAPPNAME}; + my $options = "--cluster-name=$name"; + + $options .= ' -b' if ($params{binary_start}); + # Note: We set the cluster_name here, not in postgresql.conf (in # sub init) so that it does not get copied to standbys. $ret = TestLib::system_log('pg_ctl', '-D', $self->data_dir, '-l', - $self->logfile, '-o', "--cluster-name=$name", 'start'); + $self->logfile, '-o', $options, 'start'); } if ($ret != 0) diff --git a/src/test/regress/expected/collate.icu.utf8.out b/src/test/regress/expected/collate.icu.utf8.out index 60d9263a2f..68fe86cf89 100644 --- a/src/test/regress/expected/collate.icu.utf8.out +++ b/src/test/regress/expected/collate.icu.utf8.out @@ -1897,6 +1897,193 @@ SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); t (1 row) +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); +CREATE TABLE collate_test ( + id integer, + val text COLLATE "fr-x-icu", + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); +CREATE INDEX icuidx00_val ON collate_test(val); +-- shouldn't get duplicated dependencies +CREATE INDEX icuidx00_val_val ON collate_test(val, val); +-- shouldn't track version +CREATE INDEX icuidx00_val_pattern ON collate_test(val text_pattern_ops); +-- should have single dependency, no version tracked +CREATE INDEX icuidx00_val_pattern_val_pattern ON collate_test(val text_pattern_ops, val text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_val ON collate_test(val text_pattern_ops, val); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_val_pattern ON collate_test(val, val text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_where ON collate_test(val text_pattern_ops) WHERE val >= val; +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_where ON collate_test(val) WHERE val >= val; +-- should ideally have single dependency, no version tracked, but expression walker will find a dependency on the collation and will ask to track the version +CREATE INDEX icuidx00_val_pattern_expr_pattern ON collate_test(val varchar_pattern_ops, (val || val) text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_expr ON collate_test(val varchar_pattern_ops, (val || val)); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' COLLATE custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' COLLATE custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' COLLATE "fr-x-icu"; +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + objid | refobjid | version +-----------------------------------+------------+--------------------- + icuidx00_val | "fr-x-icu" | up to date + icuidx00_val_val | "fr-x-icu" | up to date + icuidx00_val_pattern | "fr-x-icu" | version not tracked + icuidx00_val_pattern_val_pattern | "fr-x-icu" | version not tracked + icuidx00_val_pattern_val | "fr-x-icu" | up to date + icuidx00_val_val_pattern | "fr-x-icu" | up to date + icuidx00_val_pattern_where | "fr-x-icu" | up to date + icuidx00_val_where | "fr-x-icu" | up to date + icuidx00_val_pattern_expr_pattern | "fr-x-icu" | up to date + icuidx00_val_pattern_expr | "fr-x-icu" | up to date + icuidx01_t_en_fr__d_es | "en-x-icu" | up to date + icuidx01_t_en_fr__d_es | "es-x-icu" | up to date + icuidx01_t_en_fr__d_es | "fr-x-icu" | up to date + icuidx02_d_en_fr | "en-x-icu" | up to date + icuidx02_d_en_fr | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "en-x-icu" | up to date + icuidx03_t_en_fr_ga | "fr-x-icu" | up to date + icuidx03_t_en_fr_ga | "ga-x-icu" | up to date + icuidx04_d_en_fr_ga | "en-x-icu" | up to date + icuidx04_d_en_fr_ga | "fr-x-icu" | up to date + icuidx04_d_en_fr_ga | "ga-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "en-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "fr-x-icu" | up to date + icuidx05_d_en_fr_ga_arr | "ga-x-icu" | up to date + icuidx06_d_en_fr_ga | "default" | up to date + icuidx06_d_en_fr_ga | "en-x-icu" | up to date + icuidx06_d_en_fr_ga | "fr-x-icu" | up to date + icuidx06_d_en_fr_ga | "ga-x-icu" | up to date + icuidx07_d_en_fr_ga | "default" | up to date + icuidx07_d_en_fr_ga | "en-x-icu" | up to date + icuidx07_d_en_fr_ga | "fr-x-icu" | up to date + icuidx07_d_en_fr_ga | "ga-x-icu" | up to date + icuidx08_d_en_fr_ga | "en-x-icu" | up to date + icuidx08_d_en_fr_ga | "fr-x-icu" | up to date + icuidx08_d_en_fr_ga | "ga-x-icu" | up to date + icuidx09_d_en_fr_ga | "en-x-icu" | up to date + icuidx09_d_en_fr_ga | "fr-x-icu" | up to date + icuidx09_d_en_fr_ga | "ga-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "en-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "es-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "fr-x-icu" | up to date + icuidx10_d_en_fr_ga_es | "ga-x-icu" | up to date + icuidx11_d_es | "default" | up to date + icuidx11_d_es | "es-x-icu" | up to date + icuidx12_custom | "default" | up to date + icuidx12_custom | custom | up to date + icuidx13_custom | "default" | up to date + icuidx13_custom | custom | up to date + icuidx14_myrange | "default" | up to date + icuidx15_myrange_en_fr_ga | "en-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "fr-x-icu" | up to date + icuidx15_myrange_en_fr_ga | "ga-x-icu" | up to date + icuidx16_mood | "fr-x-icu" | up to date + icuidx17_part | "en-x-icu" | up to date + icuidx18_hash_d_es | "es-x-icu" | version not tracked + icuidx19_hash_id_d_es_eq | "default" | up to date + icuidx19_hash_id_d_es_eq | "es-x-icu" | up to date + icuidx20_hash_id_d_es_lt | "default" | up to date + icuidx20_hash_id_d_es_lt | "es-x-icu" | up to date +(59 rows) + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + objid +------- +(0 rows) + -- cleanup RESET search_path; SET client_min_messages TO warning; diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out index 6ace7662ee..6cb7786e13 100644 --- a/src/test/regress/expected/create_index.out +++ b/src/test/regress/expected/create_index.out @@ -2065,14 +2065,16 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(8 rows) +(10 rows) REINDEX INDEX CONCURRENTLY concur_reindex_ind1; REINDEX TABLE CONCURRENTLY concur_reindex_tab; @@ -2092,14 +2094,16 @@ WHERE classid = 'pg_class'::regclass AND obj | objref | deptype ------------------------------------------+------------------------------------------------------------+--------- index concur_reindex_ind1 | constraint concur_reindex_ind1 on table concur_reindex_tab | i + index concur_reindex_ind2 | collation "default" | n index concur_reindex_ind2 | column c2 of table concur_reindex_tab | a index concur_reindex_ind3 | column c1 of table concur_reindex_tab | a index concur_reindex_ind3 | table concur_reindex_tab | a + index concur_reindex_ind4 | collation "default" | n index concur_reindex_ind4 | column c1 of table concur_reindex_tab | a index concur_reindex_ind4 | column c2 of table concur_reindex_tab | a materialized view concur_reindex_matview | schema public | n table concur_reindex_tab | schema public | n -(8 rows) +(10 rows) -- Check that comments are preserved CREATE TABLE testcomment (i int); diff --git a/src/test/regress/sql/collate.icu.utf8.sql b/src/test/regress/sql/collate.icu.utf8.sql index 35acf91fbf..fbe220df41 100644 --- a/src/test/regress/sql/collate.icu.utf8.sql +++ b/src/test/regress/sql/collate.icu.utf8.sql @@ -716,6 +716,132 @@ INSERT INTO test33 VALUES (2, 'DEF'); -- they end up in the same partition (but it's platform-dependent which one) SELECT (SELECT count(*) FROM test33_0) <> (SELECT count(*) FROM test33_1); +-- collation versioning support +CREATE TYPE t_en_fr AS (fr text COLLATE "fr-x-icu", en text COLLATE "en-x-icu"); +CREATE DOMAIN d_en_fr AS t_en_fr; +CREATE DOMAIN d_es AS text COLLATE "es-x-icu"; +CREATE TYPE t_en_fr_ga AS (en_fr t_en_fr, ga text COLLATE "ga-x-icu"); +CREATE DOMAIN d_en_fr_ga AS t_en_fr_ga; +CREATE TYPE t_custom AS (meh text, meh2 text); +CREATE DOMAIN d_custom AS t_custom; + +CREATE COLLATION custom ( + LOCALE = 'fr-x-icu', PROVIDER = 'icu' +); + +CREATE TYPE myrange AS range (subtype = text); +CREATE TYPE myrange_en_fr_ga AS range(subtype = t_en_fr_ga); + +CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); + +CREATE TABLE collate_test ( + id integer, + val text COLLATE "fr-x-icu", + t_en_fr t_en_fr, + d_en_fr d_en_fr, + d_es d_es, + t_en_fr_ga t_en_fr_ga, + d_en_fr_ga d_en_fr_ga, + d_en_fr_ga_arr d_en_fr_ga[], + myrange myrange, + myrange_en_fr_ga myrange_en_fr_ga, + mood mood +); + +CREATE INDEX icuidx00_val ON collate_test(val); +-- shouldn't get duplicated dependencies +CREATE INDEX icuidx00_val_val ON collate_test(val, val); +-- shouldn't track version +CREATE INDEX icuidx00_val_pattern ON collate_test(val text_pattern_ops); +-- should have single dependency, no version tracked +CREATE INDEX icuidx00_val_pattern_val_pattern ON collate_test(val text_pattern_ops, val text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_val ON collate_test(val text_pattern_ops, val); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_val_pattern ON collate_test(val, val text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_where ON collate_test(val text_pattern_ops) WHERE val >= val; +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_where ON collate_test(val) WHERE val >= val; +-- should ideally have single dependency, no version tracked, but expression walker will find a dependency on the collation and will ask to track the version +CREATE INDEX icuidx00_val_pattern_expr_pattern ON collate_test(val varchar_pattern_ops, (val || val) text_pattern_ops); +-- should have single dependency, with version tracked +CREATE INDEX icuidx00_val_pattern_expr ON collate_test(val varchar_pattern_ops, (val || val)); +CREATE INDEX icuidx01_t_en_fr__d_es ON collate_test (t_en_fr, d_es); +CREATE INDEX icuidx02_d_en_fr ON collate_test (d_en_fr); +CREATE INDEX icuidx03_t_en_fr_ga ON collate_test (t_en_fr_ga); +CREATE INDEX icuidx04_d_en_fr_ga ON collate_test (d_en_fr_ga); +CREATE INDEX icuidx05_d_en_fr_ga_arr ON collate_test (d_en_fr_ga_arr); +CREATE INDEX icuidx06_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).en_fr.fr = 'foo'; +CREATE INDEX icuidx07_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga).ga = 'foo'; +CREATE INDEX icuidx08_d_en_fr_ga ON collate_test(id) WHERE (t_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx09_d_en_fr_ga ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz'); +CREATE INDEX icuidx10_d_en_fr_ga_es ON collate_test(id) WHERE (d_en_fr_ga) = ('foo', 'bar', 'baz' COLLATE "es-x-icu"); +CREATE INDEX icuidx11_d_es ON collate_test(id) WHERE (d_es) = ('foo'); +CREATE INDEX icuidx12_custom ON collate_test(id) WHERE ('foo', 'bar')::d_custom = ('foo', 'bar' COLLATE custom)::d_custom; +CREATE INDEX icuidx13_custom ON collate_test(id) WHERE ('foo' COLLATE custom, 'bar')::d_custom = ('foo', 'bar')::d_custom; +CREATE INDEX icuidx14_myrange ON collate_test(myrange); +CREATE INDEX icuidx15_myrange_en_fr_ga ON collate_test USING gist (myrange_en_fr_ga); +CREATE INDEX icuidx16_mood ON collate_test(id) WHERE mood > 'ok' COLLATE "fr-x-icu"; + +CREATE TABLE collate_part(id integer, val text COLLATE "en-x-icu") PARTITION BY range(id); +CREATE TABLE collate_part_0 PARTITION OF collate_part FOR VALUES FROM (0) TO (1); +CREATE TABLE collate_part_1 PARTITION OF collate_part FOR VALUES FROM (1) TO (1000000); +CREATE INDEX icuidx17_part ON collate_part_1 (val); +-- for key columns, hash indexes should record dependency on the collation but +-- not the version +CREATE INDEX icuidx18_hash_d_es ON collate_test USING hash (d_es); +CREATE INDEX icuidx19_hash_id_d_es_eq ON collate_test USING hash (id) WHERE (d_es) = 'foo'; +CREATE INDEX icuidx20_hash_id_d_es_lt ON collate_test USING hash (id) WHERE (d_es) < 'foo'; + +SELECT objid::regclass, refobjid::regcollation, +CASE +WHEN refobjversion IS NULL THEN 'version not tracked' +ELSE CASE + WHEN refobjversion = pg_collation_actual_version(refobjid) THEN 'up to date' + ELSE 'out of date' + END +END AS version +FROM pg_depend d +LEFT JOIN pg_class c ON c.oid = d.objid +WHERE refclassid = 'pg_collation'::regclass +AND coalesce(relkind, 'i') = 'i' +AND relname LIKE 'icuidx%' +ORDER BY 1, 2; + +-- Validate that REINDEX will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; + +REINDEX TABLE collate_test; +REINDEX TABLE collate_part_0; +REINDEX TABLE collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that REINDEX CONCURRENTLY will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +REINDEX TABLE CONCURRENTLY collate_test; +REINDEX TABLE CONCURRENTLY collate_part_0; +REINDEX INDEX CONCURRENTLY icuidx17_part; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; + +-- Validate that VACUUM FULL will update the stored version. +UPDATE pg_depend SET refobjversion = 'not a version' +WHERE refclassid = 'pg_collation'::regclass +AND objid::regclass::text LIKE 'icuidx%' +AND refobjversion IS NOT NULL; +VACUUM FULL collate_test; +VACUUM FULL collate_part_0; +VACUUM FULL collate_part_1; + +SELECT objid::regclass FROM pg_depend WHERE refobjversion = 'not a version'; -- cleanup RESET search_path; -- 2.20.1 --vy36zrg552ym33jw Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v30-0005-Add-ALTER-INDEX-.-ALTER-COLLATION-.-REFRESH-VERS.patch" ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-10 04:37 Michael Paquier <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Michael Paquier @ 2023-11-10 04:37 UTC (permalink / raw) To: David Steele <[email protected]>; +Cc: pgsql-hackers On Tue, Nov 07, 2023 at 05:20:27PM +0900, Michael Paquier wrote: > On Mon, Nov 06, 2023 at 05:39:02PM -0400, David Steele wrote: > I've retested today, and miss the failure. I'll let you know if I see > this again. I've done a few more dozen runs, and still nothing. I am wondering what this disturbance was. >> If we set these fields where backup_label was renamed, the logic would not >> be exactly the same since pg_control won't be updated until the next time >> through the loop. Since the fields should be updated before >> UpdateControlFile() I thought it made sense to keep all the updates >> together. >> >> Overall I think it is simpler, and we don't need to acquire a lock on >> ControlFile. > > What you are proposing is the same as what we already do for > backupEndRequired or backupStartPoint in the control file when > initializing recovery, so objection withdrawn. > >> Thomas included this change in his pg_basebackup changes so I did the same. >> Maybe wait a bit before we split this out? Seems like a pretty small >> change... > > Seems like a pretty good argument for refactoring that now, and let > any other patches rely on it. Would you like to send a separate > patch? The split still looks worth doing seen from here, so I am switching the patch as WoA for now. >>> Actually, grouping backupStartPointTLI and minRecoveryPointTLI should >>> reduce more the size with some alignment magic, no? >> >> I thought about this, but it seemed to me that existing fields had been >> positioned to make the grouping logical rather than to optimize alignment, >> e.g. minRecoveryPointTLI. Ideally that would have been placed near >> backupEndRequired (or vice versa). But if the general opinion is to >> rearrange for alignment, I'm OK with that. > > I've not tested, but it looks like moving backupStartPointTLI after > backupEndPoint should shave 8 bytes, if you want to maintain a more > coherent group for the LSNs. + * backupFromStandby indicates that the backup was taken on a standby. It is + * require to initialize recovery and set to false afterwards. s/require/required/. The term "backup recovery", that we've never used in the tree until now as far as I know. Perhaps this recovery method should just be referred as "recovery from backup"? By the way, there is another thing that this patch has forgotten: the SQL functions that display data from the control file. Shouldn't pg_control_recovery() be extended with the new fields? These fields may be less critical than the other ones related to recovery, but I suspect that showing them can become handy at least for debugging and monitoring purposes. Something in this area is that backupRecoveryRequired is the switch controlling if the fields set by the recovery initialization. Could it be actually useful to leave the other fields as they are and only reset backupRecoveryRequired before the first control file update? This would leave a trace of the backup history directly in the control file. What about pg_resetwal and RewriteControlFile()? Shouldn't these recovery fields be reset as well? git diff --check is complaining a bit. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-10 18:55 David Steele <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: David Steele @ 2023-11-10 18:55 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: pgsql-hackers On 11/10/23 00:37, Michael Paquier wrote: > On Tue, Nov 07, 2023 at 05:20:27PM +0900, Michael Paquier wrote: >> On Mon, Nov 06, 2023 at 05:39:02PM -0400, David Steele wrote: >> I've retested today, and miss the failure. I'll let you know if I see >> this again. > > I've done a few more dozen runs, and still nothing. I am wondering > what this disturbance was. OK, hopefully it was just a blip. >>> If we set these fields where backup_label was renamed, the logic would not >>> be exactly the same since pg_control won't be updated until the next time >>> through the loop. Since the fields should be updated before >>> UpdateControlFile() I thought it made sense to keep all the updates >>> together. >>> >>> Overall I think it is simpler, and we don't need to acquire a lock on >>> ControlFile. >> >> What you are proposing is the same as what we already do for >> backupEndRequired or backupStartPoint in the control file when >> initializing recovery, so objection withdrawn. >> >>> Thomas included this change in his pg_basebackup changes so I did the same. >>> Maybe wait a bit before we split this out? Seems like a pretty small >>> change... >> >> Seems like a pretty good argument for refactoring that now, and let >> any other patches rely on it. Would you like to send a separate >> patch? > > The split still looks worth doing seen from here, so I am switching > the patch as WoA for now. This has been split out. >>>> Actually, grouping backupStartPointTLI and minRecoveryPointTLI should >>>> reduce more the size with some alignment magic, no? >>> >>> I thought about this, but it seemed to me that existing fields had been >>> positioned to make the grouping logical rather than to optimize alignment, >>> e.g. minRecoveryPointTLI. Ideally that would have been placed near >>> backupEndRequired (or vice versa). But if the general opinion is to >>> rearrange for alignment, I'm OK with that. >> >> I've not tested, but it looks like moving backupStartPointTLI after >> backupEndPoint should shave 8 bytes, if you want to maintain a more >> coherent group for the LSNs. OK, I have moved backupStartPointTLI. > + * backupFromStandby indicates that the backup was taken on a standby. It is > + * require to initialize recovery and set to false afterwards. > s/require/required/. Fixed. > The term "backup recovery", that we've never used in the tree until > now as far as I know. Perhaps this recovery method should just be > referred as "recovery from backup"? Well, "backup recovery" is less awkward, I think. For instance "backup recovery field" vs "recovery from backup field". > By the way, there is another thing that this patch has forgotten: the > SQL functions that display data from the control file. Shouldn't > pg_control_recovery() be extended with the new fields? These fields > may be less critical than the other ones related to recovery, but I > suspect that showing them can become handy at least for debugging and > monitoring purposes. I guess that depends on whether we reset them or not (discussion below). Right now they would not be visible since by the time the user could log on they would be reset. > Something in this area is that backupRecoveryRequired is the switch > controlling if the fields set by the recovery initialization. Could > it be actually useful to leave the other fields as they are and only > reset backupRecoveryRequired before the first control file update? > This would leave a trace of the backup history directly in the control > file. Since the other recovery fields are cleared in ReachedEndOfBackup() this would be a change from what we do now. None of these fields are ever visible (with the exception of minRecoveryPoint/TLI) since they are reset when the database becomes consistent and before logons are allowed. Viewing them with pg_controldata makes sense, but I'm not sure pg_control_recovery() does. In fact, are backup_start_lsn, backup_end_lsn, and end_of_backup_record_required ever non-zero when logged onto Postgres? Maybe I'm missing something? > What about pg_resetwal and RewriteControlFile()? Shouldn't these > recovery fields be reset as well? Done. > git diff --check is complaining a bit. Fixed. New patches attached based on eb81e8e790. Regards, -David From d241a0e8aa589b3abf66331f8a3af0aabe87c214 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Fri, 10 Nov 2023 17:50:54 +0000 Subject: Allow content size to be passed to sendFileWithContent(). sendFileWithContent() previously got the content length by using strlen(), but it is also possible to pass binary content. Use len == -1 to indicate that strlen() should be use to get the content length, otherwise honor the value in len. --- src/backend/backup/basebackup.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index b537f462197..f216b588422 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -94,7 +94,7 @@ static bool verify_page_checksum(Page page, XLogRecPtr start_lsn, BlockNumber blkno, uint16 *expected_checksum); static void sendFileWithContent(bbsink *sink, const char *filename, - const char *content, + const char *content, int len, backup_manifest_info *manifest); static int64 _tarWriteHeader(bbsink *sink, const char *filename, const char *linktarget, struct stat *statbuf, @@ -334,14 +334,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) /* In the main tar, include the backup_label first... */ backup_label = build_backup_content(backup_state, false); sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, &manifest); + backup_label, -1, &manifest); pfree(backup_label); /* Then the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, - tablespace_map->data, &manifest); + tablespace_map->data, -1, &manifest); sendtblspclinks = false; } @@ -601,7 +601,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) * complete segment. */ StatusFilePath(pathbuf, walFileName, ".done"); - sendFileWithContent(sink, pathbuf, "", &manifest); + sendFileWithContent(sink, pathbuf, "", -1, &manifest); } /* @@ -629,7 +629,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) /* unconditionally mark file as archived */ StatusFilePath(pathbuf, fname, ".done"); - sendFileWithContent(sink, pathbuf, "", &manifest); + sendFileWithContent(sink, pathbuf, "", -1, &manifest); } /* Properly terminate the tar file. */ @@ -1040,22 +1040,22 @@ SendBaseBackup(BaseBackupCmd *cmd) */ static void sendFileWithContent(bbsink *sink, const char *filename, const char *content, - backup_manifest_info *manifest) + int len, backup_manifest_info *manifest) { struct stat statbuf; - int bytes_done = 0, - len; + int bytes_done = 0; pg_checksum_context checksum_ctx; if (pg_checksum_init(&checksum_ctx, manifest->checksum_type) < 0) elog(ERROR, "could not initialize checksum of file \"%s\"", filename); - len = strlen(content); + /* If len less than zero treat content as a string */ + if (len < 0) + len = strlen(content); /* - * Construct a stat struct for the backup_label file we're injecting in - * the tar. + * Construct a stat struct for the file we're injecting in the tar. */ /* Windows doesn't have the concept of uid and gid */ #ifdef WIN32 -- 2.34.1 From 79c33f300dd5bb9aabd08f27d2e6bb5857190524 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Fri, 10 Nov 2023 17:50:55 +0000 Subject: Add recovery to pg_control and remove backup_label. Simplify and harden recovery by getting rid of backup_label and storing recovery information directly in pg_control. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful restore (while almost certainly causing corruption). If pg_control is removed the cluster will not start. The user may try pg_resetwal, but I think that tool makes it pretty clear that corruption will result from its use. We could also modify pg_resetwal to complain if recovery info is present in pg_control. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves [2] without needing to write pg_control via a temp file, which may affect performance on a standby. Unfortunately, this solution cannot be back patched. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. Since backup_label is now gone, the fields that used to be in backup_label are now provided as columns returned from pg_backup_start() and pg_backup_stop() and the backup history file is still written to the archive. --- doc/src/sgml/backup.sgml | 31 ++- doc/src/sgml/func.sgml | 39 ++- doc/src/sgml/ref/pg_rewind.sgml | 3 +- src/backend/access/transam/xlog.c | 64 ++--- src/backend/access/transam/xlogbackup.c | 39 ++- src/backend/access/transam/xlogfuncs.c | 50 ++-- src/backend/access/transam/xlogrecovery.c | 250 ++++--------------- src/backend/backup/basebackup.c | 35 +-- src/backend/catalog/system_functions.sql | 10 +- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 15 +- src/bin/pg_controldata/pg_controldata.c | 8 + src/bin/pg_resetwal/pg_resetwal.c | 4 + src/bin/pg_rewind/filemap.c | 7 +- src/bin/pg_rewind/pg_rewind.c | 62 +---- src/include/access/xlog.h | 2 - src/include/access/xlogbackup.h | 14 +- src/include/access/xlogrecovery.h | 3 +- src/include/catalog/pg_control.h | 18 +- src/include/catalog/pg_proc.dat | 10 +- 19 files changed, 268 insertions(+), 396 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 8cb24d6ae54..584384875be 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -935,19 +935,20 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); ready to archive. </para> <para> - <function>pg_backup_stop</function> will return one row with three - values. The second of these fields should be written to a file named - <filename>backup_label</filename> in the root directory of the backup. The - third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <function>pg_backup_stop</function> returns the + <filename>pg_control</filename> file, which must be stored in the + <filename>global</filename> directory of the backup. It also returns the + <filename>tablespace_map</filename> file, which should be written in the + root directory of the backup unless the field is empty. These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> <para> Once the WAL segment files active during the backup are archived, you are - done. The file identified by <function>pg_backup_stop</function>'s first return + done. The file identified by <function>pg_backup_stop</function>'s + <parameter>lsn</parameter> return value is the last segment that is required to form a complete set of backup files. On a primary, if <varname>archive_mode</varname> is enabled and the <literal>wait_for_archive</literal> parameter is <literal>true</literal>, @@ -1013,7 +1014,15 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You must exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>pg_control_file</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This file contains the information + required to safely recover. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if @@ -1062,11 +1071,11 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - The backup label - file includes the label string you gave to <function>pg_backup_start</function>, + The backup history file (which is archived like WAL) includes the label + string you gave to <function>pg_backup_start</function>, as well as the time at which <function>pg_backup_start</function> was run, and the name of the starting WAL file. In case of confusion it is therefore - possible to look inside a backup file and determine exactly which + possible to look inside a backup history file and determine exactly which backup session the dump file came from. The tablespace map file includes the symbolic link names as they exist in the directory <filename>pg_tblspc/</filename> and the full path of each symbolic link. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index d963f0a0a00..ed3e5b9dce6 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26845,7 +26845,10 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <parameter>label</parameter> <type>text</type> <optional>, <parameter>fast</parameter> <type>boolean</type> </optional> ) - <returnvalue>pg_lsn</returnvalue> + <returnvalue>record</returnvalue> + ( <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>start</parameter> <type>timestamptz</type> ) </para> <para> Prepares the server to begin an on-line backup. The only required @@ -26857,6 +26860,13 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 as possible. This forces an immediate checkpoint which will cause a spike in I/O operations, slowing any concurrently executing queries. </para> + <para> + The result columns contain information about the start of the backup + and can be ignored: the <parameter>lsn</parameter> column holds the + starting write-ahead log location, the + <parameter>timeline_id</parameter> column holds the starting timeline, + and the <parameter>stop</parameter> column holds the starting timestamp. + </para> <para> This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function. @@ -26872,13 +26882,15 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <optional><parameter>wait_for_archive</parameter> <type>boolean</type> </optional> ) <returnvalue>record</returnvalue> - ( <parameter>lsn</parameter> <type>pg_lsn</type>, - <parameter>labelfile</parameter> <type>text</type>, - <parameter>spcmapfile</parameter> <type>text</type> ) + ( <parameter>pg_control_file</parameter> <type>text</type>, + <parameter>tablespace_map_file</parameter> <type>text</type>, + <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>stop</parameter> <type>timestamptz</type> ) </para> <para> Finishes performing an on-line backup. The desired contents of the - backup label file and the tablespace map file are returned as part of + pg_control file and the tablespace map file are returned as part of the result of the function and must be written to files in the backup area. These files must not be written to the live data directory (doing so will cause PostgreSQL to fail to restart in the event of a @@ -26910,13 +26922,16 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 backup. </para> <para> - The result of the function is a single record. - The <parameter>lsn</parameter> column holds the backup's ending - write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be - stored as part of the backup and are required as part of the restore - process. + The result of the function is a single record. The first column returns + the contents of the <filename>pg_control</filename> file and the + second column returns the contents of the + <filename>tablespace_map</filename> file. These must be stored as part + of the backup and are required as part of the restore process. The + remainder of the columns contain information about the end of the backup + and can be ignored: the <parameter>lsn</parameter> column holds the + ending write-ahead log location, the <parameter>timeline_id</parameter> + column holds the ending timeline, and the <parameter>stop</parameter> + column holds the ending timestamp. </para> <para> This function is restricted to superusers by default, but other users diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 8e0000d39fb..889add4c5e4 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -400,7 +400,6 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b <filename>pg_serial/</filename>, <filename>pg_snapshots/</filename>, <filename>pg_stat_tmp/</filename>, and <filename>pg_subtrans/</filename> are omitted from the data copied from the source cluster. The files - <filename>backup_label</filename>, <filename>tablespace_map</filename>, <filename>pg_internal.init</filename>, <filename>postmaster.opts</filename>, and @@ -410,7 +409,7 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b </step> <step> <para> - Create a <filename>backup_label</filename> file to begin WAL replay at + Update <filename>pg_control</filename> file to begin WAL replay at the checkpoint created at failover and configure the <filename>pg_control</filename> file with a minimum consistency LSN defined as the result of <literal>pg_current_wal_insert_lsn()</literal> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 1159dff1a69..1689bc7d3a7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -74,6 +74,7 @@ #include "pg_trace.h" #include "pgstat.h" #include "port/atomics.h" +#include "port/pg_crc32c.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" #include "postmaster/startup.h" @@ -5135,7 +5136,6 @@ StartupXLOG(void) bool wasShutdown; bool didCrash; bool haveTblspcMap; - bool haveBackupLabel; XLogRecPtr EndOfLog; TimeLineID EndOfLogTLI; TimeLineID newTLI; @@ -5259,13 +5259,14 @@ StartupXLOG(void) /* * Prepare for WAL recovery if needed. * - * InitWalRecovery analyzes the control file and the backup label file, if - * any. It updates the in-memory ControlFile buffer according to the - * starting checkpoint, and sets InRecovery and ArchiveRecoveryRequested. + * InitWalRecovery analyzes the control file and checks if backup recovery + * has been requested. It updates the in-memory ControlFile buffer + * according to the starting checkpoint, and sets InRecovery and + * ArchiveRecoveryRequested. + * * It also applies the tablespace map file, if any. */ - InitWalRecovery(ControlFile, &wasShutdown, - &haveBackupLabel, &haveTblspcMap); + InitWalRecovery(ControlFile, &wasShutdown, &haveTblspcMap); checkPoint = ControlFile->checkPointCopy; /* initialize shared memory variables from the checkpoint record */ @@ -5408,20 +5409,6 @@ StartupXLOG(void) */ UpdateControlFile(); - /* - * If there was a backup label file, it's done its job and the info - * has now been propagated into pg_control. We must get rid of the - * label file so that if we crash during recovery, we'll pick up at - * the latest recovery restartpoint instead of going all the way back - * to the backup start point. It seems prudent though to just rename - * the file out of the way rather than delete it completely. - */ - if (haveBackupLabel) - { - unlink(BACKUP_LABEL_OLD); - durable_rename(BACKUP_LABEL_FILE, BACKUP_LABEL_OLD, FATAL); - } - /* * If there was a tablespace_map file, it's done its job and the * symlinks have been created. We must get rid of the map file so @@ -5571,10 +5558,8 @@ StartupXLOG(void) * (at which point we reset backupStartPoint to be Invalid), for * backup-from-replica (which can't inject records into the WAL stream), * that point is when we reach the minRecoveryPoint in pg_control (which - * we purposefully copy last when backing up from a replica). For - * pg_rewind (which creates a backup_label with a method of "pg_rewind") - * or snapshot-style backups (which don't), backupEndRequired will be set - * to false. + * we purposefully copy last when backing up). For pg_rewind or + * snapshot-style backups, backupEndRequired will be set to false. * * Note: it is indeed okay to look at the local variable * LocalMinRecoveryPoint here, even though ControlFile->minRecoveryPoint @@ -8744,11 +8729,33 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) int seconds_before_warning; int waits = 0; bool reported_waiting = false; + ControlFileData *controlFileCopy = (ControlFileData *)state->controlFile; Assert(state != NULL); backup_stopped_in_recovery = RecoveryInProgress(); + /* + * Create a copy of control data and update it with fields required for + * recovery. Also recalculate the CRC. + */ + memset(controlFileCopy, 0, PG_CONTROL_MAX_SAFE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupRecoveryRequired = true; + controlFileCopy->backupFromStandby = backup_stopped_in_recovery; + controlFileCopy->backupEndRequired = true; + controlFileCopy->backupCheckPoint = state->checkpointloc; + controlFileCopy->backupStartPoint = state->startpoint; + controlFileCopy->backupStartPointTLI = state->starttli; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. @@ -8850,11 +8857,8 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) "Enable full_page_writes and run CHECKPOINT on the primary, " "and then try an online backup again."))); - - LWLockAcquire(ControlFileLock, LW_SHARED); - state->stoppoint = ControlFile->minRecoveryPoint; - state->stoptli = ControlFile->minRecoveryPointTLI; - LWLockRelease(ControlFileLock); + state->stoppoint = controlFileCopy->minRecoveryPoint; + state->stoptli = controlFileCopy->minRecoveryPointTLI; } else { @@ -8896,7 +8900,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) histfilepath))); /* Build and save the contents of the backup history file */ - history_file = build_backup_content(state, true); + history_file = build_backup_history_content(state); fprintf(fp, "%s", history_file); pfree(history_file); diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c index 21d68133ae1..22c95f3c4c9 100644 --- a/src/backend/access/transam/xlogbackup.c +++ b/src/backend/access/transam/xlogbackup.c @@ -18,19 +18,19 @@ #include "access/xlogbackup.h" /* - * Build contents for backup_label or backup history file. - * - * When ishistoryfile is true, it creates the contents for a backup history - * file, otherwise it creates contents for a backup_label file. + * Build contents for backup history file. * * Returns the result generated as a palloc'd string. */ char * -build_backup_content(BackupState *state, bool ishistoryfile) +build_backup_history_content(BackupState *state) { char startstrbuf[128]; + char stopstrfbuf[128]; char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */ + char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ XLogSegNo startsegno; + XLogSegNo stopsegno; StringInfo result = makeStringInfo(); char *data; @@ -45,16 +45,10 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n", LSN_FORMAT_ARGS(state->startpoint), startxlogfile); - if (ishistoryfile) - { - char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ - XLogSegNo stopsegno; - - XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); - XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); - appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", - LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); - } + XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); + XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); + appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", + LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n", LSN_FORMAT_ARGS(state->checkpointloc)); @@ -65,17 +59,12 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "LABEL: %s\n", state->name); appendStringInfo(result, "START TIMELINE: %u\n", state->starttli); - if (ishistoryfile) - { - char stopstrfbuf[128]; - - /* Use the log timezone here, not the session timezone */ - pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", - pg_localtime(&state->stoptime, log_timezone)); + /* Use the log timezone here, not the session timezone */ + pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", + pg_localtime(&state->stoptime, log_timezone)); - appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); - appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); - } + appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); + appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); data = result->data; pfree(result); diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 45a70668b1c..2388a60a5e5 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -53,7 +53,7 @@ static MemoryContext backupcontext = NULL; * pg_backup_start: set up for taking an on-line backup dump * * Essentially what this does is to create the contents required for the - * backup_label file and the tablespace map. + * the tablespace map. * * Permission checking for this function is managed through the normal * GRANT system. @@ -61,6 +61,10 @@ static MemoryContext backupcontext = NULL; Datum pg_backup_start(PG_FUNCTION_ARGS) { +#define PG_BACKUP_START_V2_COLS 3 + TupleDesc tupdesc; + Datum values[PG_BACKUP_START_V2_COLS] = {0}; + bool nulls[PG_BACKUP_START_V2_COLS] = {0}; text *backupid = PG_GETARG_TEXT_PP(0); bool fast = PG_GETARG_BOOL(1); char *backupidstr; @@ -69,6 +73,10 @@ pg_backup_start(PG_FUNCTION_ARGS) backupidstr = text_to_cstring(backupid); + /* Initialize attributes information in the tuple descriptor */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + if (status == SESSION_BACKUP_RUNNING) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -102,7 +110,12 @@ pg_backup_start(PG_FUNCTION_ARGS) register_persistent_abort_backup_handler(); do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map); - PG_RETURN_LSN(backup_state->startpoint); + values[0] = LSNGetDatum(backup_state->startpoint); + values[1] = Int64GetDatum(backup_state->starttli); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->starttime)); + + /* Returns the record as Datum */ + PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); } @@ -113,14 +126,12 @@ pg_backup_start(PG_FUNCTION_ARGS) * allows the user to choose if they want to wait for the WAL to be archived * or if we should just return as soon as the WAL record is written. * - * This function stops an in-progress backup, creates backup_label contents and - * it returns the backup stop LSN, backup_label and tablespace_map contents. + * This function stops an in-progress backup and returns the backup stop LSN, + * pg_control and tablespace_map contents. * - * The backup_label contains the user-supplied label string (typically this - * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * The pg_control file contains the recovery information for the backup. It is + * the caller's responsibility to write the pg_control and tablespace_map files + * in the data folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -128,12 +139,12 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 5 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); - char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -152,15 +163,16 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); - /* Build the contents of backup_label */ - backup_label = build_backup_content(backup_state, false); - - values[0] = LSNGetDatum(backup_state->stoppoint); - values[1] = CStringGetTextDatum(backup_label); - values[2] = CStringGetTextDatum(tablespace_map->data); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_MAX_SAFE_SIZE); - /* Deallocate backup-related variables */ - pfree(backup_label); + values[0] = PointerGetDatum(pg_control_bytea); + values[1] = CStringGetTextDatum(tablespace_map->data); + values[2] = LSNGetDatum(backup_state->stoppoint); + values[3] = Int64GetDatum(backup_state->stoptli); + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->stoptime)); /* Clean up the session-level state and its memory context */ backup_state = NULL; diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c61566666aa..f43ea39f963 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -6,7 +6,7 @@ * This source file contains functions controlling WAL recovery. * InitWalRecovery() initializes the system for crash or archive recovery, * or standby mode, depending on configuration options and the state of - * the control file and possible backup label file. PerformWalRecovery() + * the control file and possible backup recovery. PerformWalRecovery() * performs the actual WAL replay, calling the rmgr-specific redo routines. * FinishWalRecovery() performs end-of-recovery checks and cleanup actions, * and prepares information needed to initialize the WAL for writes. In @@ -152,11 +152,12 @@ static bool recovery_signal_file_found = false; /* * CheckPointLoc is the position of the checkpoint record that determines - * where to start the replay. It comes from the backup label file or the - * control file. + * where to start the replay. It comes from the control file, either from the + * default location or from a backup recovery field. * - * RedoStartLSN is the checkpoint's REDO location, also from the backup label - * file or the control file. In standby mode, XLOG streaming usually starts + * RedoStartLSN is the checkpoint's REDO location, also from the default + * control file location or from a backup recovery field. In standby mode, + * XLOG streaming usually starts * from the position where an invalid record was found. But if we fail to * read even the initial checkpoint record, we use the REDO location instead * of the checkpoint location as the start position of XLOG streaming. @@ -388,9 +389,6 @@ static void ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, Time static void EnableStandbyMode(void); static void readRecoverySignalFile(void); static void validateRecoveryParameters(void); -static bool read_backup_label(XLogRecPtr *checkPointLoc, - TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby); static bool read_tablespace_map(List **tablespaces); static void xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTLI); @@ -492,8 +490,8 @@ EnableStandbyMode(void) * Prepare the system for WAL recovery, if needed. * * This is called by StartupXLOG() which coordinates the server startup - * sequence. This function analyzes the control file and the backup label - * file, if any, and figures out whether we need to perform crash recovery or + * sequence. This function analyzes the control file and backup recovery + * info, if any, and figures out whether we need to perform crash recovery or * archive recovery, and how far we need to replay the WAL to reach a * consistent state. * @@ -510,7 +508,7 @@ EnableStandbyMode(void) */ void InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, - bool *haveBackupLabel_ptr, bool *haveTblspcMap_ptr) + bool *haveTblspcMap_ptr) { XLogPageReadPrivate *private; struct stat st; @@ -518,7 +516,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, XLogRecord *record; DBState dbstate_at_startup; bool haveTblspcMap = false; - bool haveBackupLabel = false; + bool backupRecoveryRequired = false; CheckPoint checkPoint; bool backupFromStandby = false; @@ -549,7 +547,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Set the WAL reading processor now, as it will be needed when reading - * the checkpoint record required (backup_label or not). + * the checkpoint record required (backup recovery required or not). */ private = palloc0(sizeof(XLogPageReadPrivate)); xlogreader = @@ -585,18 +583,34 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, primary_image_masked = (char *) palloc(BLCKSZ); /* - * Read the backup_label file. We want to run this part of the recovery - * process after checking for signal files and after performing validation - * of the recovery parameters. + * Load recovery settings from pg_control. We want to run this part of the + * recovery process after checking for signal files and after performing + * validation of the recovery parameters. */ - if (read_backup_label(&CheckPointLoc, &CheckPointTLI, &backupEndRequired, - &backupFromStandby)) + if (ControlFile->backupRecoveryRequired) { List *tablespaces = NIL; + /* Initialize recovery from fields stored in pg_control */ + CheckPointLoc = ControlFile->backupCheckPoint; + CheckPointTLI = ControlFile->backupStartPointTLI; + RedoStartLSN = ControlFile->backupStartPoint; + RedoStartTLI = ControlFile->backupStartPointTLI; + backupEndRequired = ControlFile->backupEndRequired; + backupFromStandby = ControlFile->backupFromStandby; + + /* Clear fields used to initialize recovery */ + ControlFile->backupCheckPoint = InvalidXLogRecPtr; + ControlFile->backupStartPointTLI = 0; + ControlFile->backupRecoveryRequired = false; + ControlFile->backupFromStandby = false; + + /* Indicate that recovery was requested */ + backupRecoveryRequired = true; + /* - * Archive recovery was requested, and thanks to the backup label - * file, we know how far we need to replay to reach consistency. Enter + * Archive recovery was requested, and thanks to the recovery + * info, we know how far we need to replay to reach consistency. Enter * archive recovery directly. */ InArchiveRecovery = true; @@ -604,8 +618,9 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, EnableStandbyMode(); /* - * When a backup_label file is present, we want to roll forward from - * the checkpoint it identifies, rather than using pg_control. + * When backup recovery is requested, we want to roll forward from + * the checkpoint it identifies, rather than using the default + * checkpoint. */ record = ReadCheckpointRecord(xlogprefetcher, CheckPointLoc, CheckPointTLI); @@ -620,9 +635,8 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Make sure that REDO location exists. This may not be the case - * if there was a crash during an online backup, which left a - * backup_label around that references a WAL segment that's - * already been archived. + * if recovery.signal is missing and the WAL has already been + * archived. */ if (checkPoint.redo < CheckPointLoc) { @@ -631,20 +645,16 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, checkPoint.ThisTimeLineID)) ereport(FATAL, (errmsg("could not find redo location referenced by checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); } } else { ereport(FATAL, (errmsg("could not locate required checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); wasShutdown = false; /* keep compiler quiet */ } @@ -679,37 +689,32 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* tell the caller to delete it later */ haveTblspcMap = true; } - - /* tell the caller to delete it later */ - haveBackupLabel = true; } else { - /* No backup_label file has been found if we are here. */ - /* - * If tablespace_map file is present without backup_label file, there - * is no use of such file. There is no harm in retaining it, but it - * is better to get rid of the map file so that we don't have any + * If tablespace_map file is present without backup recovery requested, + * there is no use of such file. There is no harm in retaining it, but + * it is better to get rid of the map file so that we don't have any * redundant file in data directory and it will avoid any sort of * confusion. It seems prudent though to just rename the file out of * the way rather than delete it completely, also we ignore any error * that occurs in rename operation as even if map file is present - * without backup_label file, it is harmless. + * without backup recovery requested, it is harmless. */ if (stat(TABLESPACE_MAP, &st) == 0) { unlink(TABLESPACE_MAP_OLD); if (durable_rename(TABLESPACE_MAP, TABLESPACE_MAP_OLD, DEBUG1) == 0) ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("File \"%s\" was renamed to \"%s\".", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); else ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("Could not rename file \"%s\" to \"%s\": %m.", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); } @@ -943,7 +948,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. */ - if (haveBackupLabel) + if (backupRecoveryRequired) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; @@ -953,7 +958,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY && dbstate_at_startup != DB_SHUTDOWNED_IN_RECOVERY) ereport(FATAL, - (errmsg("backup_label contains data inconsistent with control file"), + (errmsg("pg_control contains inconsistent data for standby backup"), errhint("This means that the backup is corrupted and you will " "have to use another backup for recovery."))); ControlFile->backupEndPoint = ControlFile->minRecoveryPoint; @@ -983,7 +988,6 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, missingContrecPtr = InvalidXLogRecPtr; *wasShutdown_ptr = wasShutdown; - *haveBackupLabel_ptr = haveBackupLabel; *haveTblspcMap_ptr = haveTblspcMap; } @@ -1156,154 +1160,6 @@ validateRecoveryParameters(void) } } -/* - * read_backup_label: check to see if a backup_label file is present - * - * If we see a backup_label during recovery, we assume that we are recovering - * from a backup dump file, and we therefore roll forward from the checkpoint - * identified by the label file, NOT what pg_control says. This avoids the - * problem that pg_control might have been archived one or more checkpoints - * later than the start of the dump, and so if we rely on it as the start - * point, we will fail to restore a consistent database state. - * - * Returns true if a backup_label was found (and fills the checkpoint - * location and TLI into *checkPointLoc and *backupLabelTLI, respectively); - * returns false if not. If this backup_label came from a streamed backup, - * *backupEndRequired is set to true. If this backup_label was created during - * recovery, *backupFromStandby is set to true. - * - * Also sets the global variables RedoStartLSN and RedoStartTLI with the LSN - * and TLI read from the backup file. - */ -static bool -read_backup_label(XLogRecPtr *checkPointLoc, TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby) -{ - char startxlogfilename[MAXFNAMELEN]; - TimeLineID tli_from_walseg, - tli_from_file; - FILE *lfp; - char ch; - char backuptype[20]; - char backupfrom[20]; - char backuplabel[MAXPGPATH]; - char backuptime[128]; - uint32 hi, - lo; - - /* suppress possible uninitialized-variable warnings */ - *checkPointLoc = InvalidXLogRecPtr; - *backupLabelTLI = 0; - *backupEndRequired = false; - *backupFromStandby = false; - - /* - * See if label file is present - */ - lfp = AllocateFile(BACKUP_LABEL_FILE, "r"); - if (!lfp) - { - if (errno != ENOENT) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - return false; /* it's not there, all is fine */ - } - - /* - * Read and parse the START WAL LOCATION and CHECKPOINT lines (this code - * is pretty crude, but we are not expecting any variability in the file - * format). - */ - if (fscanf(lfp, "START WAL LOCATION: %X/%X (file %08X%16s)%c", - &hi, &lo, &tli_from_walseg, startxlogfilename, &ch) != 5 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - RedoStartLSN = ((uint64) hi) << 32 | lo; - RedoStartTLI = tli_from_walseg; - if (fscanf(lfp, "CHECKPOINT LOCATION: %X/%X%c", - &hi, &lo, &ch) != 3 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - *checkPointLoc = ((uint64) hi) << 32 | lo; - *backupLabelTLI = tli_from_walseg; - - /* - * BACKUP METHOD lets us know if this was a typical backup ("streamed", - * which could mean either pg_basebackup or the pg_backup_start/stop - * method was used) or if this label came from somewhere else (the only - * other option today being from pg_rewind). If this was a streamed - * backup then we know that we need to play through until we get to the - * end of the WAL which was generated during the backup (at which point we - * will have reached consistency and backupEndRequired will be reset to be - * false). - */ - if (fscanf(lfp, "BACKUP METHOD: %19s\n", backuptype) == 1) - { - if (strcmp(backuptype, "streamed") == 0) - *backupEndRequired = true; - } - - /* - * BACKUP FROM lets us know if this was from a primary or a standby. If - * it was from a standby, we'll double-check that the control file state - * matches that of a standby. - */ - if (fscanf(lfp, "BACKUP FROM: %19s\n", backupfrom) == 1) - { - if (strcmp(backupfrom, "standby") == 0) - *backupFromStandby = true; - } - - /* - * Parse START TIME and LABEL. Those are not mandatory fields for recovery - * but checking for their presence is useful for debugging and the next - * sanity checks. Cope also with the fact that the result buffers have a - * pre-allocated size, hence if the backup_label file has been generated - * with strings longer than the maximum assumed here an incorrect parsing - * happens. That's fine as only minor consistency checks are done - * afterwards. - */ - if (fscanf(lfp, "START TIME: %127[^\n]\n", backuptime) == 1) - ereport(DEBUG1, - (errmsg_internal("backup time %s in file \"%s\"", - backuptime, BACKUP_LABEL_FILE))); - - if (fscanf(lfp, "LABEL: %1023[^\n]\n", backuplabel) == 1) - ereport(DEBUG1, - (errmsg_internal("backup label %s in file \"%s\"", - backuplabel, BACKUP_LABEL_FILE))); - - /* - * START TIMELINE is new as of 11. Its parsing is not mandatory, still use - * it as a sanity check if present. - */ - if (fscanf(lfp, "START TIMELINE: %u\n", &tli_from_file) == 1) - { - if (tli_from_walseg != tli_from_file) - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE), - errdetail("Timeline ID parsed is %u, but expected %u.", - tli_from_file, tli_from_walseg))); - - ereport(DEBUG1, - (errmsg_internal("backup timeline %u in file \"%s\"", - tli_from_file, BACKUP_LABEL_FILE))); - } - - if (ferror(lfp) || FreeFile(lfp)) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - - return true; -} - /* * read_tablespace_map: check to see if a tablespace_map file is present * diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index f216b588422..a4bd79447fd 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -22,6 +22,7 @@ #include "backup/basebackup.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "commands/defrem.h" #include "common/compression.h" #include "common/file_perm.h" @@ -192,10 +193,9 @@ static const struct exclude_list_item excludeFiles[] = {RELCACHE_INIT_FILENAME, true}, /* - * backup_label and tablespace_map should not exist in a running cluster - * capable of doing an online backup, but exclude them just in case. + * tablespace_map should not exist in a running cluster capable of doing + * an online backup, but exclude it just in case. */ - {BACKUP_LABEL_FILE, false}, {TABLESPACE_MAP, false}, /* @@ -325,19 +325,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; - char *backup_label; bbsink_begin_archive(sink, "base.tar"); - /* In the main tar, include the backup_label first... */ - backup_label = build_backup_content(backup_state, false); - sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, -1, &manifest); - pfree(backup_label); - - /* Then the tablespace_map file, if required... */ + /* Send the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, @@ -349,14 +341,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid); - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, &manifest); + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + + /* Send copy of pg_control containing recovery info */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_MAX_SAFE_SIZE, &manifest); } else { @@ -390,9 +382,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) } } - basebackup_progress_wait_wal_archive(&state); - do_pg_backup_stop(backup_state, !opt->nowait); - endptr = backup_state->stoppoint; endtli = backup_state->stoptli; diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 35d738d5763..24bf34b45eb 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -384,13 +384,15 @@ BEGIN ATOMIC END; CREATE OR REPLACE FUNCTION - pg_backup_start(label text, fast boolean DEFAULT false) - RETURNS pg_lsn STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' + pg_backup_start(label text, fast boolean DEFAULT false, OUT lsn pg_lsn, + OUT timeline_id int8, OUT start timestamptz) + RETURNS record STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' PARALLEL RESTRICTED; CREATE OR REPLACE FUNCTION pg_backup_stop ( - wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + wait_for_archive boolean DEFAULT true, OUT pg_control_file bytea, + OUT tablespace_map_file text, OUT lsn pg_lsn, OUT timeline_id int8, + OUT stop timestamptz) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index b9f5e1266b4..c655cb03352 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -171,8 +171,8 @@ SKIP: # Write some files to test that they are not copied. foreach my $filename ( - qw(backup_label tablespace_map postgresql.auto.conf.tmp - current_logfiles.tmp global/pg_internal.init.123)) + qw(tablespace_map postgresql.auto.conf.tmp current_logfiles.tmp + global/pg_internal.init.123)) { open my $file, '>>', "$pgdata/$filename"; print $file "DONOTCOPY"; @@ -261,14 +261,13 @@ foreach my $filename (@tempRelationFiles) "base/$postgresOid/$filename not copied"); } -# Make sure existing backup_label was ignored. -isnt(slurp_file("$tempdir/backup/backup_label"), - 'DONOTCOPY', 'existing backup_label not copied'); +# Make sure existing tablespace_map was ignored. +ok(!-f "$tempdir/backup/tablespace_map", 'tablespace_map not in backup'); rmtree("$tempdir/backup"); -# Now delete the bogus backup_label file since it will interfere with startup -unlink("$pgdata/backup_label") - or BAIL_OUT("unable to unlink $pgdata/backup_label"); +# Now delete the bogus tablespace_map file since it will interfere with startup +unlink("$pgdata/tablespace_map") + or BAIL_OUT("unable to unlink $pgdata/tablespace_map"); $node->command_ok( [ diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93e0837947c..cc515b622ff 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -277,10 +277,18 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->minRecoveryPoint)); printf(_("Min recovery ending loc's timeline: %u\n"), ControlFile->minRecoveryPointTLI); + printf(_("Backup checkpoint location: %X/%X\n"), + LSN_FORMAT_ARGS(ControlFile->backupCheckPoint)); printf(_("Backup start location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupStartPoint)); + printf(_("Backup start location's timeline: %u\n"), + ControlFile->backupStartPointTLI); printf(_("Backup end location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); + printf(_("Backup recovery required: %s\n"), + ControlFile->backupRecoveryRequired ? _("yes") : _("no")); + printf(_("Backup from standby: %s\n"), + ControlFile->backupFromStandby ? _("yes") : _("no")); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 3ae3fc06df2..255101ff3a1 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -870,8 +870,12 @@ RewriteControlFile(void) ControlFile.checkPoint = ControlFile.checkPointCopy.redo; ControlFile.minRecoveryPoint = 0; ControlFile.minRecoveryPointTLI = 0; + ControlFile.backupCheckPoint = 0; ControlFile.backupStartPoint = 0; + ControlFile.backupStartPointTLI = 0; ControlFile.backupEndPoint = 0; + ControlFile.backupRecoveryRequired = false; + ControlFile.backupFromStandby = false; ControlFile.backupEndRequired = false; /* diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index ecadd69dc53..213f4e71b88 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -139,11 +139,10 @@ static const struct exclude_list_item excludeFiles[] = {"pg_internal.init", true}, /* defined as RELCACHE_INIT_FILENAME */ /* - * If there is a backup_label or tablespace_map file, it indicates that a - * recovery failed and this cluster probably can't be rewound, but exclude - * them anyway if they are found. + * If there is a tablespace_map file, it indicates that a recovery failed + * and this cluster probably can't be rewound, but exclude it anyway if it + * is found. */ - {"backup_label", false}, /* defined as BACKUP_LABEL_FILE */ {"tablespace_map", false}, /* defined as TABLESPACE_MAP */ /* diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index bfd44a284e2..f42782e2eab 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -39,9 +39,6 @@ static void perform_rewind(filemap_t *filemap, rewind_source *source, TimeLineID chkpttli, XLogRecPtr chkptredo); -static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, - XLogRecPtr checkpointloc); - static void digestControlFile(ControlFileData *ControlFile, const char *content, size_t size); static void getRestoreCommand(const char *argv0); @@ -654,7 +651,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, pg_log_info("creating backup label and updating control file"); /* - * Create a backup label file, to tell the target where to begin the WAL + * Get recovery fields to tell the target where to begin the WAL * replay. Normally, from the last common checkpoint between the source * and the target. But if the source is a standby server, it's possible * that the last common checkpoint is *after* the standby's restartpoint. @@ -672,7 +669,6 @@ perform_rewind(filemap_t *filemap, rewind_source *source, chkpttli = ControlFile_source.checkPointCopy.ThisTimeLineID; chkptrec = ControlFile_source.checkPoint; } - createBackupLabel(chkptredo, chkpttli, chkptrec); /* * Update control file of target, to tell the target how far it must @@ -722,6 +718,12 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupRecoveryRequired = true; + ControlFile_new.backupFromStandby = true; + ControlFile_new.backupEndRequired = false; + ControlFile_new.backupCheckPoint = chkptrec; + ControlFile_new.backupStartPoint = chkptredo; + ControlFile_new.backupStartPointTLI = chkpttli; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } @@ -729,7 +731,10 @@ perform_rewind(filemap_t *filemap, rewind_source *source, static void sanityChecks(void) { - /* TODO Check that there's no backup_label in either cluster */ + /* + * TODO Check that neither cluster has backupRecoveryRequested set in + * pg_control. + */ /* Check system_identifier match */ if (ControlFile_target.system_identifier != ControlFile_source.system_identifier) @@ -951,51 +956,6 @@ findCommonAncestorTimeline(TimeLineHistoryEntry *a_history, int a_nentries, } } - -/* - * Create a backup_label file that forces recovery to begin at the last common - * checkpoint. - */ -static void -createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, XLogRecPtr checkpointloc) -{ - XLogSegNo startsegno; - time_t stamp_time; - char strfbuf[128]; - char xlogfilename[MAXFNAMELEN]; - struct tm *tmp; - char buf[1000]; - int len; - - XLByteToSeg(startpoint, startsegno, WalSegSz); - XLogFileName(xlogfilename, starttli, startsegno, WalSegSz); - - /* - * Construct backup label file - */ - stamp_time = time(NULL); - tmp = localtime(&stamp_time); - strftime(strfbuf, sizeof(strfbuf), "%Y-%m-%d %H:%M:%S %Z", tmp); - - len = snprintf(buf, sizeof(buf), - "START WAL LOCATION: %X/%X (file %s)\n" - "CHECKPOINT LOCATION: %X/%X\n" - "BACKUP METHOD: pg_rewind\n" - "BACKUP FROM: standby\n" - "START TIME: %s\n", - /* omit LABEL: line */ - LSN_FORMAT_ARGS(startpoint), xlogfilename, - LSN_FORMAT_ARGS(checkpointloc), - strfbuf); - if (len >= sizeof(buf)) - pg_fatal("backup label buffer too small"); /* shouldn't happen */ - - /* TODO: move old file out of the way, if any. */ - open_target_file("backup_label", true); /* BACKUP_LABEL_FILE */ - write_target_range(buf, 0, len); - close_target_file(); -} - /* * Check CRC of control file */ diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index a14126d164f..3aac6839a70 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -293,8 +293,6 @@ extern SessionBackupState get_backup_status(void); /* File path names (all relative to $PGDATA) */ #define RECOVERY_SIGNAL_FILE "recovery.signal" #define STANDBY_SIGNAL_FILE "standby.signal" -#define BACKUP_LABEL_FILE "backup_label" -#define BACKUP_LABEL_OLD "backup_label.old" #define TABLESPACE_MAP "tablespace_map" #define TABLESPACE_MAP_OLD "tablespace_map.old" diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h index 1611358137b..f2c3672fed6 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" /* Structure to hold backup state. */ @@ -33,9 +34,18 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. Fields have been + * updated for recovery and the CRC has been recalculated. The buffer + * is padded to PG_CONTROL_MAX_SAFE_SIZE so that pg_control is always + * a consistent size but smaller (and hopefully easier to handle) than + * PG_CONTROL_FILE_SIZE. Bytes after sizeof(ControlFileData) are zeroed. + */ + uint8_t controlFile[PG_CONTROL_MAX_SAFE_SIZE]; } BackupState; -extern char *build_backup_content(BackupState *state, - bool ishistoryfile); +extern char *build_backup_history_content(BackupState *state); #endif /* XLOG_BACKUP_H */ diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrecovery.h index ee0bc742782..981266f7340 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -80,8 +80,7 @@ extern Size XLogRecoveryShmemSize(void); extern void XLogRecoveryShmemInit(void); extern void InitWalRecovery(ControlFileData *ControlFile, - bool *wasShutdown_ptr, bool *haveBackupLabel_ptr, - bool *haveTblspcMap_ptr); + bool *wasShutdown_ptr, bool *haveTblspcMap_ptr); extern void PerformWalRecovery(void); /* diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 2ae72e3b266..8144c972ec1 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -146,6 +146,9 @@ typedef struct ControlFileData * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * + * backupCheckPoint is the backup start checkpoint and is set to zero after + * recovery is initialized. + * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and @@ -160,14 +163,27 @@ typedef struct ControlFileData * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * + * backupRecoveryRequired indicates that the pg_control file was provided + * by a backup or pg_rewind and recovery settings need to be copied. It will + * be set to false when the settings have been copied. + * + * backupFromStandby indicates that the backup was taken on a standby. It is + * required to initialize recovery and set to false afterwards. + * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely - * start up. + * start up. Currently backupEndRequired should only be false if recovery + * settings were configured by pg_rewind, which does not require an end + * point. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; + XLogRecPtr backupCheckPoint; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; + TimeLineID backupStartPointTLI; + bool backupRecoveryRequired; + bool backupFromStandby; bool backupEndRequired; /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f14aed422a7..cc8156c57e7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6413,13 +6413,17 @@ prosrc => 'pg_terminate_backend' }, { oid => '2172', descr => 'prepare for taking an online backup', proname => 'pg_backup_start', provolatile => 'v', proparallel => 'r', - prorettype => 'pg_lsn', proargtypes => 'text bool', + prorettype => 'record', proargtypes => 'text bool', + proallargtypes => '{text,bool,pg_lsn,int8,timestamptz}', + proargmodes => '{i,i,o,o,o}', + proargnames => '{label,fast,lsn,timeline_id,start}', prosrc => 'pg_backup_start' }, { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,bytea,text,pg_lsn,int8,timestamptz}', + proargmodes => '{i,o,o,o,o,o}', + proargnames => '{wait_for_archive,pg_control_file,tablespace_map_file,lsn,timeline_id,stop}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', -- 2.34.1 Attachments: [text/plain] recovery-in-pgcontrol-v4-0001-pass-len-to-sendFileWithContent.patch (3.3K, ../../[email protected]/2-recovery-in-pgcontrol-v4-0001-pass-len-to-sendFileWithContent.patch) download | inline diff: From d241a0e8aa589b3abf66331f8a3af0aabe87c214 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Fri, 10 Nov 2023 17:50:54 +0000 Subject: Allow content size to be passed to sendFileWithContent(). sendFileWithContent() previously got the content length by using strlen(), but it is also possible to pass binary content. Use len == -1 to indicate that strlen() should be use to get the content length, otherwise honor the value in len. --- src/backend/backup/basebackup.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index b537f462197..f216b588422 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -94,7 +94,7 @@ static bool verify_page_checksum(Page page, XLogRecPtr start_lsn, BlockNumber blkno, uint16 *expected_checksum); static void sendFileWithContent(bbsink *sink, const char *filename, - const char *content, + const char *content, int len, backup_manifest_info *manifest); static int64 _tarWriteHeader(bbsink *sink, const char *filename, const char *linktarget, struct stat *statbuf, @@ -334,14 +334,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) /* In the main tar, include the backup_label first... */ backup_label = build_backup_content(backup_state, false); sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, &manifest); + backup_label, -1, &manifest); pfree(backup_label); /* Then the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, - tablespace_map->data, &manifest); + tablespace_map->data, -1, &manifest); sendtblspclinks = false; } @@ -601,7 +601,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) * complete segment. */ StatusFilePath(pathbuf, walFileName, ".done"); - sendFileWithContent(sink, pathbuf, "", &manifest); + sendFileWithContent(sink, pathbuf, "", -1, &manifest); } /* @@ -629,7 +629,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) /* unconditionally mark file as archived */ StatusFilePath(pathbuf, fname, ".done"); - sendFileWithContent(sink, pathbuf, "", &manifest); + sendFileWithContent(sink, pathbuf, "", -1, &manifest); } /* Properly terminate the tar file. */ @@ -1040,22 +1040,22 @@ SendBaseBackup(BaseBackupCmd *cmd) */ static void sendFileWithContent(bbsink *sink, const char *filename, const char *content, - backup_manifest_info *manifest) + int len, backup_manifest_info *manifest) { struct stat statbuf; - int bytes_done = 0, - len; + int bytes_done = 0; pg_checksum_context checksum_ctx; if (pg_checksum_init(&checksum_ctx, manifest->checksum_type) < 0) elog(ERROR, "could not initialize checksum of file \"%s\"", filename); - len = strlen(content); + /* If len less than zero treat content as a string */ + if (len < 0) + len = strlen(content); /* - * Construct a stat struct for the backup_label file we're injecting in - * the tar. + * Construct a stat struct for the file we're injecting in the tar. */ /* Windows doesn't have the concept of uid and gid */ #ifdef WIN32 -- 2.34.1 [text/plain] recovery-in-pgcontrol-v4-0002-remove-backuplabel.patch (57.8K, ../../[email protected]/3-recovery-in-pgcontrol-v4-0002-remove-backuplabel.patch) download | inline diff: From 79c33f300dd5bb9aabd08f27d2e6bb5857190524 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Fri, 10 Nov 2023 17:50:55 +0000 Subject: Add recovery to pg_control and remove backup_label. Simplify and harden recovery by getting rid of backup_label and storing recovery information directly in pg_control. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful restore (while almost certainly causing corruption). If pg_control is removed the cluster will not start. The user may try pg_resetwal, but I think that tool makes it pretty clear that corruption will result from its use. We could also modify pg_resetwal to complain if recovery info is present in pg_control. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves [2] without needing to write pg_control via a temp file, which may affect performance on a standby. Unfortunately, this solution cannot be back patched. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. Since backup_label is now gone, the fields that used to be in backup_label are now provided as columns returned from pg_backup_start() and pg_backup_stop() and the backup history file is still written to the archive. --- doc/src/sgml/backup.sgml | 31 ++- doc/src/sgml/func.sgml | 39 ++- doc/src/sgml/ref/pg_rewind.sgml | 3 +- src/backend/access/transam/xlog.c | 64 ++--- src/backend/access/transam/xlogbackup.c | 39 ++- src/backend/access/transam/xlogfuncs.c | 50 ++-- src/backend/access/transam/xlogrecovery.c | 250 ++++--------------- src/backend/backup/basebackup.c | 35 +-- src/backend/catalog/system_functions.sql | 10 +- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 15 +- src/bin/pg_controldata/pg_controldata.c | 8 + src/bin/pg_resetwal/pg_resetwal.c | 4 + src/bin/pg_rewind/filemap.c | 7 +- src/bin/pg_rewind/pg_rewind.c | 62 +---- src/include/access/xlog.h | 2 - src/include/access/xlogbackup.h | 14 +- src/include/access/xlogrecovery.h | 3 +- src/include/catalog/pg_control.h | 18 +- src/include/catalog/pg_proc.dat | 10 +- 19 files changed, 268 insertions(+), 396 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 8cb24d6ae54..584384875be 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -935,19 +935,20 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); ready to archive. </para> <para> - <function>pg_backup_stop</function> will return one row with three - values. The second of these fields should be written to a file named - <filename>backup_label</filename> in the root directory of the backup. The - third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <function>pg_backup_stop</function> returns the + <filename>pg_control</filename> file, which must be stored in the + <filename>global</filename> directory of the backup. It also returns the + <filename>tablespace_map</filename> file, which should be written in the + root directory of the backup unless the field is empty. These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> <para> Once the WAL segment files active during the backup are archived, you are - done. The file identified by <function>pg_backup_stop</function>'s first return + done. The file identified by <function>pg_backup_stop</function>'s + <parameter>lsn</parameter> return value is the last segment that is required to form a complete set of backup files. On a primary, if <varname>archive_mode</varname> is enabled and the <literal>wait_for_archive</literal> parameter is <literal>true</literal>, @@ -1013,7 +1014,15 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You must exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>pg_control_file</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This file contains the information + required to safely recover. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if @@ -1062,11 +1071,11 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - The backup label - file includes the label string you gave to <function>pg_backup_start</function>, + The backup history file (which is archived like WAL) includes the label + string you gave to <function>pg_backup_start</function>, as well as the time at which <function>pg_backup_start</function> was run, and the name of the starting WAL file. In case of confusion it is therefore - possible to look inside a backup file and determine exactly which + possible to look inside a backup history file and determine exactly which backup session the dump file came from. The tablespace map file includes the symbolic link names as they exist in the directory <filename>pg_tblspc/</filename> and the full path of each symbolic link. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index d963f0a0a00..ed3e5b9dce6 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26845,7 +26845,10 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <parameter>label</parameter> <type>text</type> <optional>, <parameter>fast</parameter> <type>boolean</type> </optional> ) - <returnvalue>pg_lsn</returnvalue> + <returnvalue>record</returnvalue> + ( <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>start</parameter> <type>timestamptz</type> ) </para> <para> Prepares the server to begin an on-line backup. The only required @@ -26857,6 +26860,13 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 as possible. This forces an immediate checkpoint which will cause a spike in I/O operations, slowing any concurrently executing queries. </para> + <para> + The result columns contain information about the start of the backup + and can be ignored: the <parameter>lsn</parameter> column holds the + starting write-ahead log location, the + <parameter>timeline_id</parameter> column holds the starting timeline, + and the <parameter>stop</parameter> column holds the starting timestamp. + </para> <para> This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function. @@ -26872,13 +26882,15 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <optional><parameter>wait_for_archive</parameter> <type>boolean</type> </optional> ) <returnvalue>record</returnvalue> - ( <parameter>lsn</parameter> <type>pg_lsn</type>, - <parameter>labelfile</parameter> <type>text</type>, - <parameter>spcmapfile</parameter> <type>text</type> ) + ( <parameter>pg_control_file</parameter> <type>text</type>, + <parameter>tablespace_map_file</parameter> <type>text</type>, + <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>stop</parameter> <type>timestamptz</type> ) </para> <para> Finishes performing an on-line backup. The desired contents of the - backup label file and the tablespace map file are returned as part of + pg_control file and the tablespace map file are returned as part of the result of the function and must be written to files in the backup area. These files must not be written to the live data directory (doing so will cause PostgreSQL to fail to restart in the event of a @@ -26910,13 +26922,16 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 backup. </para> <para> - The result of the function is a single record. - The <parameter>lsn</parameter> column holds the backup's ending - write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be - stored as part of the backup and are required as part of the restore - process. + The result of the function is a single record. The first column returns + the contents of the <filename>pg_control</filename> file and the + second column returns the contents of the + <filename>tablespace_map</filename> file. These must be stored as part + of the backup and are required as part of the restore process. The + remainder of the columns contain information about the end of the backup + and can be ignored: the <parameter>lsn</parameter> column holds the + ending write-ahead log location, the <parameter>timeline_id</parameter> + column holds the ending timeline, and the <parameter>stop</parameter> + column holds the ending timestamp. </para> <para> This function is restricted to superusers by default, but other users diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 8e0000d39fb..889add4c5e4 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -400,7 +400,6 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b <filename>pg_serial/</filename>, <filename>pg_snapshots/</filename>, <filename>pg_stat_tmp/</filename>, and <filename>pg_subtrans/</filename> are omitted from the data copied from the source cluster. The files - <filename>backup_label</filename>, <filename>tablespace_map</filename>, <filename>pg_internal.init</filename>, <filename>postmaster.opts</filename>, and @@ -410,7 +409,7 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b </step> <step> <para> - Create a <filename>backup_label</filename> file to begin WAL replay at + Update <filename>pg_control</filename> file to begin WAL replay at the checkpoint created at failover and configure the <filename>pg_control</filename> file with a minimum consistency LSN defined as the result of <literal>pg_current_wal_insert_lsn()</literal> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 1159dff1a69..1689bc7d3a7 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -74,6 +74,7 @@ #include "pg_trace.h" #include "pgstat.h" #include "port/atomics.h" +#include "port/pg_crc32c.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" #include "postmaster/startup.h" @@ -5135,7 +5136,6 @@ StartupXLOG(void) bool wasShutdown; bool didCrash; bool haveTblspcMap; - bool haveBackupLabel; XLogRecPtr EndOfLog; TimeLineID EndOfLogTLI; TimeLineID newTLI; @@ -5259,13 +5259,14 @@ StartupXLOG(void) /* * Prepare for WAL recovery if needed. * - * InitWalRecovery analyzes the control file and the backup label file, if - * any. It updates the in-memory ControlFile buffer according to the - * starting checkpoint, and sets InRecovery and ArchiveRecoveryRequested. + * InitWalRecovery analyzes the control file and checks if backup recovery + * has been requested. It updates the in-memory ControlFile buffer + * according to the starting checkpoint, and sets InRecovery and + * ArchiveRecoveryRequested. + * * It also applies the tablespace map file, if any. */ - InitWalRecovery(ControlFile, &wasShutdown, - &haveBackupLabel, &haveTblspcMap); + InitWalRecovery(ControlFile, &wasShutdown, &haveTblspcMap); checkPoint = ControlFile->checkPointCopy; /* initialize shared memory variables from the checkpoint record */ @@ -5408,20 +5409,6 @@ StartupXLOG(void) */ UpdateControlFile(); - /* - * If there was a backup label file, it's done its job and the info - * has now been propagated into pg_control. We must get rid of the - * label file so that if we crash during recovery, we'll pick up at - * the latest recovery restartpoint instead of going all the way back - * to the backup start point. It seems prudent though to just rename - * the file out of the way rather than delete it completely. - */ - if (haveBackupLabel) - { - unlink(BACKUP_LABEL_OLD); - durable_rename(BACKUP_LABEL_FILE, BACKUP_LABEL_OLD, FATAL); - } - /* * If there was a tablespace_map file, it's done its job and the * symlinks have been created. We must get rid of the map file so @@ -5571,10 +5558,8 @@ StartupXLOG(void) * (at which point we reset backupStartPoint to be Invalid), for * backup-from-replica (which can't inject records into the WAL stream), * that point is when we reach the minRecoveryPoint in pg_control (which - * we purposefully copy last when backing up from a replica). For - * pg_rewind (which creates a backup_label with a method of "pg_rewind") - * or snapshot-style backups (which don't), backupEndRequired will be set - * to false. + * we purposefully copy last when backing up). For pg_rewind or + * snapshot-style backups, backupEndRequired will be set to false. * * Note: it is indeed okay to look at the local variable * LocalMinRecoveryPoint here, even though ControlFile->minRecoveryPoint @@ -8744,11 +8729,33 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) int seconds_before_warning; int waits = 0; bool reported_waiting = false; + ControlFileData *controlFileCopy = (ControlFileData *)state->controlFile; Assert(state != NULL); backup_stopped_in_recovery = RecoveryInProgress(); + /* + * Create a copy of control data and update it with fields required for + * recovery. Also recalculate the CRC. + */ + memset(controlFileCopy, 0, PG_CONTROL_MAX_SAFE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupRecoveryRequired = true; + controlFileCopy->backupFromStandby = backup_stopped_in_recovery; + controlFileCopy->backupEndRequired = true; + controlFileCopy->backupCheckPoint = state->checkpointloc; + controlFileCopy->backupStartPoint = state->startpoint; + controlFileCopy->backupStartPointTLI = state->starttli; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. @@ -8850,11 +8857,8 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) "Enable full_page_writes and run CHECKPOINT on the primary, " "and then try an online backup again."))); - - LWLockAcquire(ControlFileLock, LW_SHARED); - state->stoppoint = ControlFile->minRecoveryPoint; - state->stoptli = ControlFile->minRecoveryPointTLI; - LWLockRelease(ControlFileLock); + state->stoppoint = controlFileCopy->minRecoveryPoint; + state->stoptli = controlFileCopy->minRecoveryPointTLI; } else { @@ -8896,7 +8900,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) histfilepath))); /* Build and save the contents of the backup history file */ - history_file = build_backup_content(state, true); + history_file = build_backup_history_content(state); fprintf(fp, "%s", history_file); pfree(history_file); diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c index 21d68133ae1..22c95f3c4c9 100644 --- a/src/backend/access/transam/xlogbackup.c +++ b/src/backend/access/transam/xlogbackup.c @@ -18,19 +18,19 @@ #include "access/xlogbackup.h" /* - * Build contents for backup_label or backup history file. - * - * When ishistoryfile is true, it creates the contents for a backup history - * file, otherwise it creates contents for a backup_label file. + * Build contents for backup history file. * * Returns the result generated as a palloc'd string. */ char * -build_backup_content(BackupState *state, bool ishistoryfile) +build_backup_history_content(BackupState *state) { char startstrbuf[128]; + char stopstrfbuf[128]; char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */ + char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ XLogSegNo startsegno; + XLogSegNo stopsegno; StringInfo result = makeStringInfo(); char *data; @@ -45,16 +45,10 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n", LSN_FORMAT_ARGS(state->startpoint), startxlogfile); - if (ishistoryfile) - { - char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ - XLogSegNo stopsegno; - - XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); - XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); - appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", - LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); - } + XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); + XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); + appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", + LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n", LSN_FORMAT_ARGS(state->checkpointloc)); @@ -65,17 +59,12 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "LABEL: %s\n", state->name); appendStringInfo(result, "START TIMELINE: %u\n", state->starttli); - if (ishistoryfile) - { - char stopstrfbuf[128]; - - /* Use the log timezone here, not the session timezone */ - pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", - pg_localtime(&state->stoptime, log_timezone)); + /* Use the log timezone here, not the session timezone */ + pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", + pg_localtime(&state->stoptime, log_timezone)); - appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); - appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); - } + appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); + appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); data = result->data; pfree(result); diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 45a70668b1c..2388a60a5e5 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -53,7 +53,7 @@ static MemoryContext backupcontext = NULL; * pg_backup_start: set up for taking an on-line backup dump * * Essentially what this does is to create the contents required for the - * backup_label file and the tablespace map. + * the tablespace map. * * Permission checking for this function is managed through the normal * GRANT system. @@ -61,6 +61,10 @@ static MemoryContext backupcontext = NULL; Datum pg_backup_start(PG_FUNCTION_ARGS) { +#define PG_BACKUP_START_V2_COLS 3 + TupleDesc tupdesc; + Datum values[PG_BACKUP_START_V2_COLS] = {0}; + bool nulls[PG_BACKUP_START_V2_COLS] = {0}; text *backupid = PG_GETARG_TEXT_PP(0); bool fast = PG_GETARG_BOOL(1); char *backupidstr; @@ -69,6 +73,10 @@ pg_backup_start(PG_FUNCTION_ARGS) backupidstr = text_to_cstring(backupid); + /* Initialize attributes information in the tuple descriptor */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + if (status == SESSION_BACKUP_RUNNING) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -102,7 +110,12 @@ pg_backup_start(PG_FUNCTION_ARGS) register_persistent_abort_backup_handler(); do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map); - PG_RETURN_LSN(backup_state->startpoint); + values[0] = LSNGetDatum(backup_state->startpoint); + values[1] = Int64GetDatum(backup_state->starttli); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->starttime)); + + /* Returns the record as Datum */ + PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); } @@ -113,14 +126,12 @@ pg_backup_start(PG_FUNCTION_ARGS) * allows the user to choose if they want to wait for the WAL to be archived * or if we should just return as soon as the WAL record is written. * - * This function stops an in-progress backup, creates backup_label contents and - * it returns the backup stop LSN, backup_label and tablespace_map contents. + * This function stops an in-progress backup and returns the backup stop LSN, + * pg_control and tablespace_map contents. * - * The backup_label contains the user-supplied label string (typically this - * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * The pg_control file contains the recovery information for the backup. It is + * the caller's responsibility to write the pg_control and tablespace_map files + * in the data folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -128,12 +139,12 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 5 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); - char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -152,15 +163,16 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); - /* Build the contents of backup_label */ - backup_label = build_backup_content(backup_state, false); - - values[0] = LSNGetDatum(backup_state->stoppoint); - values[1] = CStringGetTextDatum(backup_label); - values[2] = CStringGetTextDatum(tablespace_map->data); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_MAX_SAFE_SIZE); - /* Deallocate backup-related variables */ - pfree(backup_label); + values[0] = PointerGetDatum(pg_control_bytea); + values[1] = CStringGetTextDatum(tablespace_map->data); + values[2] = LSNGetDatum(backup_state->stoppoint); + values[3] = Int64GetDatum(backup_state->stoptli); + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->stoptime)); /* Clean up the session-level state and its memory context */ backup_state = NULL; diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c61566666aa..f43ea39f963 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -6,7 +6,7 @@ * This source file contains functions controlling WAL recovery. * InitWalRecovery() initializes the system for crash or archive recovery, * or standby mode, depending on configuration options and the state of - * the control file and possible backup label file. PerformWalRecovery() + * the control file and possible backup recovery. PerformWalRecovery() * performs the actual WAL replay, calling the rmgr-specific redo routines. * FinishWalRecovery() performs end-of-recovery checks and cleanup actions, * and prepares information needed to initialize the WAL for writes. In @@ -152,11 +152,12 @@ static bool recovery_signal_file_found = false; /* * CheckPointLoc is the position of the checkpoint record that determines - * where to start the replay. It comes from the backup label file or the - * control file. + * where to start the replay. It comes from the control file, either from the + * default location or from a backup recovery field. * - * RedoStartLSN is the checkpoint's REDO location, also from the backup label - * file or the control file. In standby mode, XLOG streaming usually starts + * RedoStartLSN is the checkpoint's REDO location, also from the default + * control file location or from a backup recovery field. In standby mode, + * XLOG streaming usually starts * from the position where an invalid record was found. But if we fail to * read even the initial checkpoint record, we use the REDO location instead * of the checkpoint location as the start position of XLOG streaming. @@ -388,9 +389,6 @@ static void ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, Time static void EnableStandbyMode(void); static void readRecoverySignalFile(void); static void validateRecoveryParameters(void); -static bool read_backup_label(XLogRecPtr *checkPointLoc, - TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby); static bool read_tablespace_map(List **tablespaces); static void xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTLI); @@ -492,8 +490,8 @@ EnableStandbyMode(void) * Prepare the system for WAL recovery, if needed. * * This is called by StartupXLOG() which coordinates the server startup - * sequence. This function analyzes the control file and the backup label - * file, if any, and figures out whether we need to perform crash recovery or + * sequence. This function analyzes the control file and backup recovery + * info, if any, and figures out whether we need to perform crash recovery or * archive recovery, and how far we need to replay the WAL to reach a * consistent state. * @@ -510,7 +508,7 @@ EnableStandbyMode(void) */ void InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, - bool *haveBackupLabel_ptr, bool *haveTblspcMap_ptr) + bool *haveTblspcMap_ptr) { XLogPageReadPrivate *private; struct stat st; @@ -518,7 +516,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, XLogRecord *record; DBState dbstate_at_startup; bool haveTblspcMap = false; - bool haveBackupLabel = false; + bool backupRecoveryRequired = false; CheckPoint checkPoint; bool backupFromStandby = false; @@ -549,7 +547,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Set the WAL reading processor now, as it will be needed when reading - * the checkpoint record required (backup_label or not). + * the checkpoint record required (backup recovery required or not). */ private = palloc0(sizeof(XLogPageReadPrivate)); xlogreader = @@ -585,18 +583,34 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, primary_image_masked = (char *) palloc(BLCKSZ); /* - * Read the backup_label file. We want to run this part of the recovery - * process after checking for signal files and after performing validation - * of the recovery parameters. + * Load recovery settings from pg_control. We want to run this part of the + * recovery process after checking for signal files and after performing + * validation of the recovery parameters. */ - if (read_backup_label(&CheckPointLoc, &CheckPointTLI, &backupEndRequired, - &backupFromStandby)) + if (ControlFile->backupRecoveryRequired) { List *tablespaces = NIL; + /* Initialize recovery from fields stored in pg_control */ + CheckPointLoc = ControlFile->backupCheckPoint; + CheckPointTLI = ControlFile->backupStartPointTLI; + RedoStartLSN = ControlFile->backupStartPoint; + RedoStartTLI = ControlFile->backupStartPointTLI; + backupEndRequired = ControlFile->backupEndRequired; + backupFromStandby = ControlFile->backupFromStandby; + + /* Clear fields used to initialize recovery */ + ControlFile->backupCheckPoint = InvalidXLogRecPtr; + ControlFile->backupStartPointTLI = 0; + ControlFile->backupRecoveryRequired = false; + ControlFile->backupFromStandby = false; + + /* Indicate that recovery was requested */ + backupRecoveryRequired = true; + /* - * Archive recovery was requested, and thanks to the backup label - * file, we know how far we need to replay to reach consistency. Enter + * Archive recovery was requested, and thanks to the recovery + * info, we know how far we need to replay to reach consistency. Enter * archive recovery directly. */ InArchiveRecovery = true; @@ -604,8 +618,9 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, EnableStandbyMode(); /* - * When a backup_label file is present, we want to roll forward from - * the checkpoint it identifies, rather than using pg_control. + * When backup recovery is requested, we want to roll forward from + * the checkpoint it identifies, rather than using the default + * checkpoint. */ record = ReadCheckpointRecord(xlogprefetcher, CheckPointLoc, CheckPointTLI); @@ -620,9 +635,8 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Make sure that REDO location exists. This may not be the case - * if there was a crash during an online backup, which left a - * backup_label around that references a WAL segment that's - * already been archived. + * if recovery.signal is missing and the WAL has already been + * archived. */ if (checkPoint.redo < CheckPointLoc) { @@ -631,20 +645,16 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, checkPoint.ThisTimeLineID)) ereport(FATAL, (errmsg("could not find redo location referenced by checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); } } else { ereport(FATAL, (errmsg("could not locate required checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); wasShutdown = false; /* keep compiler quiet */ } @@ -679,37 +689,32 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* tell the caller to delete it later */ haveTblspcMap = true; } - - /* tell the caller to delete it later */ - haveBackupLabel = true; } else { - /* No backup_label file has been found if we are here. */ - /* - * If tablespace_map file is present without backup_label file, there - * is no use of such file. There is no harm in retaining it, but it - * is better to get rid of the map file so that we don't have any + * If tablespace_map file is present without backup recovery requested, + * there is no use of such file. There is no harm in retaining it, but + * it is better to get rid of the map file so that we don't have any * redundant file in data directory and it will avoid any sort of * confusion. It seems prudent though to just rename the file out of * the way rather than delete it completely, also we ignore any error * that occurs in rename operation as even if map file is present - * without backup_label file, it is harmless. + * without backup recovery requested, it is harmless. */ if (stat(TABLESPACE_MAP, &st) == 0) { unlink(TABLESPACE_MAP_OLD); if (durable_rename(TABLESPACE_MAP, TABLESPACE_MAP_OLD, DEBUG1) == 0) ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("File \"%s\" was renamed to \"%s\".", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); else ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("Could not rename file \"%s\" to \"%s\": %m.", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); } @@ -943,7 +948,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. */ - if (haveBackupLabel) + if (backupRecoveryRequired) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; @@ -953,7 +958,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY && dbstate_at_startup != DB_SHUTDOWNED_IN_RECOVERY) ereport(FATAL, - (errmsg("backup_label contains data inconsistent with control file"), + (errmsg("pg_control contains inconsistent data for standby backup"), errhint("This means that the backup is corrupted and you will " "have to use another backup for recovery."))); ControlFile->backupEndPoint = ControlFile->minRecoveryPoint; @@ -983,7 +988,6 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, missingContrecPtr = InvalidXLogRecPtr; *wasShutdown_ptr = wasShutdown; - *haveBackupLabel_ptr = haveBackupLabel; *haveTblspcMap_ptr = haveTblspcMap; } @@ -1156,154 +1160,6 @@ validateRecoveryParameters(void) } } -/* - * read_backup_label: check to see if a backup_label file is present - * - * If we see a backup_label during recovery, we assume that we are recovering - * from a backup dump file, and we therefore roll forward from the checkpoint - * identified by the label file, NOT what pg_control says. This avoids the - * problem that pg_control might have been archived one or more checkpoints - * later than the start of the dump, and so if we rely on it as the start - * point, we will fail to restore a consistent database state. - * - * Returns true if a backup_label was found (and fills the checkpoint - * location and TLI into *checkPointLoc and *backupLabelTLI, respectively); - * returns false if not. If this backup_label came from a streamed backup, - * *backupEndRequired is set to true. If this backup_label was created during - * recovery, *backupFromStandby is set to true. - * - * Also sets the global variables RedoStartLSN and RedoStartTLI with the LSN - * and TLI read from the backup file. - */ -static bool -read_backup_label(XLogRecPtr *checkPointLoc, TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby) -{ - char startxlogfilename[MAXFNAMELEN]; - TimeLineID tli_from_walseg, - tli_from_file; - FILE *lfp; - char ch; - char backuptype[20]; - char backupfrom[20]; - char backuplabel[MAXPGPATH]; - char backuptime[128]; - uint32 hi, - lo; - - /* suppress possible uninitialized-variable warnings */ - *checkPointLoc = InvalidXLogRecPtr; - *backupLabelTLI = 0; - *backupEndRequired = false; - *backupFromStandby = false; - - /* - * See if label file is present - */ - lfp = AllocateFile(BACKUP_LABEL_FILE, "r"); - if (!lfp) - { - if (errno != ENOENT) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - return false; /* it's not there, all is fine */ - } - - /* - * Read and parse the START WAL LOCATION and CHECKPOINT lines (this code - * is pretty crude, but we are not expecting any variability in the file - * format). - */ - if (fscanf(lfp, "START WAL LOCATION: %X/%X (file %08X%16s)%c", - &hi, &lo, &tli_from_walseg, startxlogfilename, &ch) != 5 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - RedoStartLSN = ((uint64) hi) << 32 | lo; - RedoStartTLI = tli_from_walseg; - if (fscanf(lfp, "CHECKPOINT LOCATION: %X/%X%c", - &hi, &lo, &ch) != 3 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - *checkPointLoc = ((uint64) hi) << 32 | lo; - *backupLabelTLI = tli_from_walseg; - - /* - * BACKUP METHOD lets us know if this was a typical backup ("streamed", - * which could mean either pg_basebackup or the pg_backup_start/stop - * method was used) or if this label came from somewhere else (the only - * other option today being from pg_rewind). If this was a streamed - * backup then we know that we need to play through until we get to the - * end of the WAL which was generated during the backup (at which point we - * will have reached consistency and backupEndRequired will be reset to be - * false). - */ - if (fscanf(lfp, "BACKUP METHOD: %19s\n", backuptype) == 1) - { - if (strcmp(backuptype, "streamed") == 0) - *backupEndRequired = true; - } - - /* - * BACKUP FROM lets us know if this was from a primary or a standby. If - * it was from a standby, we'll double-check that the control file state - * matches that of a standby. - */ - if (fscanf(lfp, "BACKUP FROM: %19s\n", backupfrom) == 1) - { - if (strcmp(backupfrom, "standby") == 0) - *backupFromStandby = true; - } - - /* - * Parse START TIME and LABEL. Those are not mandatory fields for recovery - * but checking for their presence is useful for debugging and the next - * sanity checks. Cope also with the fact that the result buffers have a - * pre-allocated size, hence if the backup_label file has been generated - * with strings longer than the maximum assumed here an incorrect parsing - * happens. That's fine as only minor consistency checks are done - * afterwards. - */ - if (fscanf(lfp, "START TIME: %127[^\n]\n", backuptime) == 1) - ereport(DEBUG1, - (errmsg_internal("backup time %s in file \"%s\"", - backuptime, BACKUP_LABEL_FILE))); - - if (fscanf(lfp, "LABEL: %1023[^\n]\n", backuplabel) == 1) - ereport(DEBUG1, - (errmsg_internal("backup label %s in file \"%s\"", - backuplabel, BACKUP_LABEL_FILE))); - - /* - * START TIMELINE is new as of 11. Its parsing is not mandatory, still use - * it as a sanity check if present. - */ - if (fscanf(lfp, "START TIMELINE: %u\n", &tli_from_file) == 1) - { - if (tli_from_walseg != tli_from_file) - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE), - errdetail("Timeline ID parsed is %u, but expected %u.", - tli_from_file, tli_from_walseg))); - - ereport(DEBUG1, - (errmsg_internal("backup timeline %u in file \"%s\"", - tli_from_file, BACKUP_LABEL_FILE))); - } - - if (ferror(lfp) || FreeFile(lfp)) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - - return true; -} - /* * read_tablespace_map: check to see if a tablespace_map file is present * diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index f216b588422..a4bd79447fd 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -22,6 +22,7 @@ #include "backup/basebackup.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "commands/defrem.h" #include "common/compression.h" #include "common/file_perm.h" @@ -192,10 +193,9 @@ static const struct exclude_list_item excludeFiles[] = {RELCACHE_INIT_FILENAME, true}, /* - * backup_label and tablespace_map should not exist in a running cluster - * capable of doing an online backup, but exclude them just in case. + * tablespace_map should not exist in a running cluster capable of doing + * an online backup, but exclude it just in case. */ - {BACKUP_LABEL_FILE, false}, {TABLESPACE_MAP, false}, /* @@ -325,19 +325,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; - char *backup_label; bbsink_begin_archive(sink, "base.tar"); - /* In the main tar, include the backup_label first... */ - backup_label = build_backup_content(backup_state, false); - sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, -1, &manifest); - pfree(backup_label); - - /* Then the tablespace_map file, if required... */ + /* Send the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, @@ -349,14 +341,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid); - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, &manifest); + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + + /* Send copy of pg_control containing recovery info */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_MAX_SAFE_SIZE, &manifest); } else { @@ -390,9 +382,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) } } - basebackup_progress_wait_wal_archive(&state); - do_pg_backup_stop(backup_state, !opt->nowait); - endptr = backup_state->stoppoint; endtli = backup_state->stoptli; diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 35d738d5763..24bf34b45eb 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -384,13 +384,15 @@ BEGIN ATOMIC END; CREATE OR REPLACE FUNCTION - pg_backup_start(label text, fast boolean DEFAULT false) - RETURNS pg_lsn STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' + pg_backup_start(label text, fast boolean DEFAULT false, OUT lsn pg_lsn, + OUT timeline_id int8, OUT start timestamptz) + RETURNS record STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' PARALLEL RESTRICTED; CREATE OR REPLACE FUNCTION pg_backup_stop ( - wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + wait_for_archive boolean DEFAULT true, OUT pg_control_file bytea, + OUT tablespace_map_file text, OUT lsn pg_lsn, OUT timeline_id int8, + OUT stop timestamptz) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index b9f5e1266b4..c655cb03352 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -171,8 +171,8 @@ SKIP: # Write some files to test that they are not copied. foreach my $filename ( - qw(backup_label tablespace_map postgresql.auto.conf.tmp - current_logfiles.tmp global/pg_internal.init.123)) + qw(tablespace_map postgresql.auto.conf.tmp current_logfiles.tmp + global/pg_internal.init.123)) { open my $file, '>>', "$pgdata/$filename"; print $file "DONOTCOPY"; @@ -261,14 +261,13 @@ foreach my $filename (@tempRelationFiles) "base/$postgresOid/$filename not copied"); } -# Make sure existing backup_label was ignored. -isnt(slurp_file("$tempdir/backup/backup_label"), - 'DONOTCOPY', 'existing backup_label not copied'); +# Make sure existing tablespace_map was ignored. +ok(!-f "$tempdir/backup/tablespace_map", 'tablespace_map not in backup'); rmtree("$tempdir/backup"); -# Now delete the bogus backup_label file since it will interfere with startup -unlink("$pgdata/backup_label") - or BAIL_OUT("unable to unlink $pgdata/backup_label"); +# Now delete the bogus tablespace_map file since it will interfere with startup +unlink("$pgdata/tablespace_map") + or BAIL_OUT("unable to unlink $pgdata/tablespace_map"); $node->command_ok( [ diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93e0837947c..cc515b622ff 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -277,10 +277,18 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->minRecoveryPoint)); printf(_("Min recovery ending loc's timeline: %u\n"), ControlFile->minRecoveryPointTLI); + printf(_("Backup checkpoint location: %X/%X\n"), + LSN_FORMAT_ARGS(ControlFile->backupCheckPoint)); printf(_("Backup start location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupStartPoint)); + printf(_("Backup start location's timeline: %u\n"), + ControlFile->backupStartPointTLI); printf(_("Backup end location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); + printf(_("Backup recovery required: %s\n"), + ControlFile->backupRecoveryRequired ? _("yes") : _("no")); + printf(_("Backup from standby: %s\n"), + ControlFile->backupFromStandby ? _("yes") : _("no")); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 3ae3fc06df2..255101ff3a1 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -870,8 +870,12 @@ RewriteControlFile(void) ControlFile.checkPoint = ControlFile.checkPointCopy.redo; ControlFile.minRecoveryPoint = 0; ControlFile.minRecoveryPointTLI = 0; + ControlFile.backupCheckPoint = 0; ControlFile.backupStartPoint = 0; + ControlFile.backupStartPointTLI = 0; ControlFile.backupEndPoint = 0; + ControlFile.backupRecoveryRequired = false; + ControlFile.backupFromStandby = false; ControlFile.backupEndRequired = false; /* diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index ecadd69dc53..213f4e71b88 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -139,11 +139,10 @@ static const struct exclude_list_item excludeFiles[] = {"pg_internal.init", true}, /* defined as RELCACHE_INIT_FILENAME */ /* - * If there is a backup_label or tablespace_map file, it indicates that a - * recovery failed and this cluster probably can't be rewound, but exclude - * them anyway if they are found. + * If there is a tablespace_map file, it indicates that a recovery failed + * and this cluster probably can't be rewound, but exclude it anyway if it + * is found. */ - {"backup_label", false}, /* defined as BACKUP_LABEL_FILE */ {"tablespace_map", false}, /* defined as TABLESPACE_MAP */ /* diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index bfd44a284e2..f42782e2eab 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -39,9 +39,6 @@ static void perform_rewind(filemap_t *filemap, rewind_source *source, TimeLineID chkpttli, XLogRecPtr chkptredo); -static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, - XLogRecPtr checkpointloc); - static void digestControlFile(ControlFileData *ControlFile, const char *content, size_t size); static void getRestoreCommand(const char *argv0); @@ -654,7 +651,7 @@ perform_rewind(filemap_t *filemap, rewind_source *source, pg_log_info("creating backup label and updating control file"); /* - * Create a backup label file, to tell the target where to begin the WAL + * Get recovery fields to tell the target where to begin the WAL * replay. Normally, from the last common checkpoint between the source * and the target. But if the source is a standby server, it's possible * that the last common checkpoint is *after* the standby's restartpoint. @@ -672,7 +669,6 @@ perform_rewind(filemap_t *filemap, rewind_source *source, chkpttli = ControlFile_source.checkPointCopy.ThisTimeLineID; chkptrec = ControlFile_source.checkPoint; } - createBackupLabel(chkptredo, chkpttli, chkptrec); /* * Update control file of target, to tell the target how far it must @@ -722,6 +718,12 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupRecoveryRequired = true; + ControlFile_new.backupFromStandby = true; + ControlFile_new.backupEndRequired = false; + ControlFile_new.backupCheckPoint = chkptrec; + ControlFile_new.backupStartPoint = chkptredo; + ControlFile_new.backupStartPointTLI = chkpttli; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } @@ -729,7 +731,10 @@ perform_rewind(filemap_t *filemap, rewind_source *source, static void sanityChecks(void) { - /* TODO Check that there's no backup_label in either cluster */ + /* + * TODO Check that neither cluster has backupRecoveryRequested set in + * pg_control. + */ /* Check system_identifier match */ if (ControlFile_target.system_identifier != ControlFile_source.system_identifier) @@ -951,51 +956,6 @@ findCommonAncestorTimeline(TimeLineHistoryEntry *a_history, int a_nentries, } } - -/* - * Create a backup_label file that forces recovery to begin at the last common - * checkpoint. - */ -static void -createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, XLogRecPtr checkpointloc) -{ - XLogSegNo startsegno; - time_t stamp_time; - char strfbuf[128]; - char xlogfilename[MAXFNAMELEN]; - struct tm *tmp; - char buf[1000]; - int len; - - XLByteToSeg(startpoint, startsegno, WalSegSz); - XLogFileName(xlogfilename, starttli, startsegno, WalSegSz); - - /* - * Construct backup label file - */ - stamp_time = time(NULL); - tmp = localtime(&stamp_time); - strftime(strfbuf, sizeof(strfbuf), "%Y-%m-%d %H:%M:%S %Z", tmp); - - len = snprintf(buf, sizeof(buf), - "START WAL LOCATION: %X/%X (file %s)\n" - "CHECKPOINT LOCATION: %X/%X\n" - "BACKUP METHOD: pg_rewind\n" - "BACKUP FROM: standby\n" - "START TIME: %s\n", - /* omit LABEL: line */ - LSN_FORMAT_ARGS(startpoint), xlogfilename, - LSN_FORMAT_ARGS(checkpointloc), - strfbuf); - if (len >= sizeof(buf)) - pg_fatal("backup label buffer too small"); /* shouldn't happen */ - - /* TODO: move old file out of the way, if any. */ - open_target_file("backup_label", true); /* BACKUP_LABEL_FILE */ - write_target_range(buf, 0, len); - close_target_file(); -} - /* * Check CRC of control file */ diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index a14126d164f..3aac6839a70 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -293,8 +293,6 @@ extern SessionBackupState get_backup_status(void); /* File path names (all relative to $PGDATA) */ #define RECOVERY_SIGNAL_FILE "recovery.signal" #define STANDBY_SIGNAL_FILE "standby.signal" -#define BACKUP_LABEL_FILE "backup_label" -#define BACKUP_LABEL_OLD "backup_label.old" #define TABLESPACE_MAP "tablespace_map" #define TABLESPACE_MAP_OLD "tablespace_map.old" diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h index 1611358137b..f2c3672fed6 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" /* Structure to hold backup state. */ @@ -33,9 +34,18 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. Fields have been + * updated for recovery and the CRC has been recalculated. The buffer + * is padded to PG_CONTROL_MAX_SAFE_SIZE so that pg_control is always + * a consistent size but smaller (and hopefully easier to handle) than + * PG_CONTROL_FILE_SIZE. Bytes after sizeof(ControlFileData) are zeroed. + */ + uint8_t controlFile[PG_CONTROL_MAX_SAFE_SIZE]; } BackupState; -extern char *build_backup_content(BackupState *state, - bool ishistoryfile); +extern char *build_backup_history_content(BackupState *state); #endif /* XLOG_BACKUP_H */ diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrecovery.h index ee0bc742782..981266f7340 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -80,8 +80,7 @@ extern Size XLogRecoveryShmemSize(void); extern void XLogRecoveryShmemInit(void); extern void InitWalRecovery(ControlFileData *ControlFile, - bool *wasShutdown_ptr, bool *haveBackupLabel_ptr, - bool *haveTblspcMap_ptr); + bool *wasShutdown_ptr, bool *haveTblspcMap_ptr); extern void PerformWalRecovery(void); /* diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 2ae72e3b266..8144c972ec1 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -146,6 +146,9 @@ typedef struct ControlFileData * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * + * backupCheckPoint is the backup start checkpoint and is set to zero after + * recovery is initialized. + * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and @@ -160,14 +163,27 @@ typedef struct ControlFileData * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * + * backupRecoveryRequired indicates that the pg_control file was provided + * by a backup or pg_rewind and recovery settings need to be copied. It will + * be set to false when the settings have been copied. + * + * backupFromStandby indicates that the backup was taken on a standby. It is + * required to initialize recovery and set to false afterwards. + * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely - * start up. + * start up. Currently backupEndRequired should only be false if recovery + * settings were configured by pg_rewind, which does not require an end + * point. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; + XLogRecPtr backupCheckPoint; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; + TimeLineID backupStartPointTLI; + bool backupRecoveryRequired; + bool backupFromStandby; bool backupEndRequired; /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f14aed422a7..cc8156c57e7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6413,13 +6413,17 @@ prosrc => 'pg_terminate_backend' }, { oid => '2172', descr => 'prepare for taking an online backup', proname => 'pg_backup_start', provolatile => 'v', proparallel => 'r', - prorettype => 'pg_lsn', proargtypes => 'text bool', + prorettype => 'record', proargtypes => 'text bool', + proallargtypes => '{text,bool,pg_lsn,int8,timestamptz}', + proargmodes => '{i,i,o,o,o}', + proargnames => '{label,fast,lsn,timeline_id,start}', prosrc => 'pg_backup_start' }, { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,bytea,text,pg_lsn,int8,timestamptz}', + proargmodes => '{i,o,o,o,o,o}', + proargnames => '{wait_for_archive,pg_control_file,tablespace_map_file,lsn,timeline_id,stop}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', -- 2.34.1 ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-13 04:36 Michael Paquier <[email protected]> parent: David Steele <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Michael Paquier @ 2023-11-13 04:36 UTC (permalink / raw) To: David Steele <[email protected]>; +Cc: pgsql-hackers On Fri, Nov 10, 2023 at 02:55:19PM -0400, David Steele wrote: > On 11/10/23 00:37, Michael Paquier wrote: >> I've done a few more dozen runs, and still nothing. I am wondering >> what this disturbance was. > > OK, hopefully it was just a blip. Still nothing on this side. So that seems really like a random blip in the matrix. > This has been split out. Thanks, applied 0001. >> The term "backup recovery", that we've never used in the tree until >> now as far as I know. Perhaps this recovery method should just be >> referred as "recovery from backup"? > > Well, "backup recovery" is less awkward, I think. For instance "backup > recovery field" vs "recovery from backup field". Not sure. I've never used this term when referring to recovery from a backup. Perhaps I'm just not used to it, still that sounds a bit confusing here. >> Something in this area is that backupRecoveryRequired is the switch >> controlling if the fields set by the recovery initialization. Could >> it be actually useful to leave the other fields as they are and only >> reset backupRecoveryRequired before the first control file update? >> This would leave a trace of the backup history directly in the control >> file. > > Since the other recovery fields are cleared in ReachedEndOfBackup() this > would be a change from what we do now. > > None of these fields are ever visible (with the exception of > minRecoveryPoint/TLI) since they are reset when the database becomes > consistent and before logons are allowed. Viewing them with pg_controldata > makes sense, but I'm not sure pg_control_recovery() does. > > In fact, are backup_start_lsn, backup_end_lsn, and > end_of_backup_record_required ever non-zero when logged onto Postgres? Maybe > I'm missing something? Yeah, but custom backup/restore tools may want manipulate the contents of the control file for their own work, so at least for the sake of visibility it sounds important to me to show all the information at hand, and that there is no need to. - The backup label - file includes the label string you gave to <function>pg_backup_start</function>, + The backup history file (which is archived like WAL) includes the label + string you gave to <function>pg_backup_start</function>, as well as the time at which <function>pg_backup_start</function> was run, and the name of the starting WAL file. In case of confusion it is therefore - possible to look inside a backup file and determine exactly which + possible to look inside a backup history file and determine exactly which As a side note, it is a bit disappointing that we lose the backup label from the backup itself, even if the patch changes correctly the documentation to reflect the new behavior. It is in the backup history file on the node from where the base backup has been taken or in the archives, hopefully. However there is nothing that remains in the base backup itself, and backups can be self-contained (easy with pg_basebackup --wal-method=stream). I think that we should retain a minimum amount of information as a replacement for the backup_label, at least. With this state, the current patch slightly reduces the debuggability of deployments. That could be annoying for some users. > New patches attached based on eb81e8e790. Diving into the code for references about the backup label file, I have spotted this log in pg_rewind that is now incorrect: if (showprogress) pg_log_info("creating backup label and updating control file"); + printf(_("Backup start location's timeline: %u\n"), + ControlFile->backupStartPointTLI); printf(_("Backup end location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); Perhaps these two should be reversed to match with the header file. + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. Fields have been + * updated for recovery and the CRC has been recalculated. The buffer + * is padded to PG_CONTROL_MAX_SAFE_SIZE so that pg_control is always + * a consistent size but smaller (and hopefully easier to handle) than + * PG_CONTROL_FILE_SIZE. Bytes after sizeof(ControlFileData) are zeroed. + */ + uint8_t controlFile[PG_CONTROL_MAX_SAFE_SIZE]; I don't mind the addition of a control file with the max safe size, because it will never be higher than that. However: + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + + /* Send copy of pg_control containing recovery info */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_MAX_SAFE_SIZE, &manifest); It seems to me that the base backup protocol should always send an 8k file for the control file so as we maintain consistency with the on-disk format. Currently, a base backup taken with this patch results in a control file of size 512B. + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_MAX_SAFE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_MAX_SAFE_SIZE); Similar comment for the control file returned by pg_backup_stop(), which could just be made a 8k field? + <function>pg_backup_stop</function> returns the + <filename>pg_control</filename> file, which must be stored in the + <filename>global</filename> directory of the backup. It also returns the And perhaps emphasize that this file should be an 8kB file in the paragraph mentioning the data returned by pg_backup_stop()? - Create a <filename>backup_label</filename> file to begin WAL replay at + Update <filename>pg_control</filename> file to begin WAL replay at the checkpoint created at failover and configure the <filename>pg_control</filename> file with a minimum consistency LSN pg_control is mentioned twice, so perhaps this could be worded better? PG_CONTROL_VERSION is important to not forget about.. Perhaps this should be noted somewhere, or just changed in the patch itself. Contrary to catalog changes, we do few of these in the control file so there is close to zero risk of conflicts with other patches in the CF app. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-20 01:15 Michael Paquier <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 3 replies; 11+ messages in thread From: Michael Paquier @ 2023-11-20 01:15 UTC (permalink / raw) To: David Steele <[email protected]>; +Cc: Robert Haas <[email protected]>; Postgres hackers <[email protected]> (I am not exactly sure how, but we've lost pgsql-hackers on the way when you sent v5. Now added back in CC with the two latest patches you've proposed attached.) Here is a short summary of what has been missed by the lists: - I've commented that the patch should not create, not show up in fields returned the SQL functions or stream control files with a size of 512B, just stick to 8kB. If this is worth changing this should be applied consistently across the board including initdb, discussed on its own thread. - The backup-related fields in the control file are reset at the end of recovery. I've suggested to not do that to keep a trace of what was happening during recovery. The latest version of the patch resets the fields. - With the backup_label file gone, we lose some information in the backups themselves, which is not good. Instead, you have suggested an approach where this data is added to the backup manifest, meaning that no information would be lost, particularly useful for self-contained backups. The fields planned to be added to the backup manifest are: -- The start and end time of the backup, the end timestamp being useful to know when stop time can be used for PITR. -- The backup label. I've agreed that it may be the best thing to do at this end to not lose any data related to the removal of the backup_label file. On Sun, Nov 19, 2023 at 02:14:32PM -0400, David Steele wrote: > On 11/15/23 20:03, Michael Paquier wrote: >> As the label is only an informational field, the parsing added to >> pg_verifybackup is not really needed because it is used nowhere in the >> validation process, so keeping the logic simpler would be the way to >> go IMO. This is contrary to the WAL range for example, where start >> and end LSNs are used for validation with a pg_waldump command. >> Robert, any comments about the addition of the label in the manifest? > > I'm sure Robert will comment on this when he gets the time, but for now I > have backed off on passing the new info to pg_verifybackup and added > start/stop time. FWIW, I'm OK with the bits for the backup manifest as presented. So if there are no remarks and/or no objections, I'd like to apply it but let give some room to others to comment on that as there's been a gap in the emails exchanged on pgsql-hackers. I hope that the summary I've posted above covers everything. So let's see about doing something around the middle of next week. With Thanksgiving in the US, a lot of folks will not have the time to monitor what's happening on this thread. + The end time for the backup. This is when the backup was stopped in + <productname>PostgreSQL</productname> and represents the earliest time + that can be used for time-based Point-In-Time Recovery. This one is actually a very good point. We'd lost this capacity with the backup_label file gone without the end timestamps in the control file. > New patches attached based on b218fbb7. I've noticed on the other thread the remark about being less aggressive with the fields related to recovery in the control file, so I assume that this patch should leave the fields be after the end of recovery from the start and only rely on backupRecoveryRequired to decide if the recovery should use the fields or not: https://www.postgresql.org/message-id/241ccde1-1928-4ba2-a0bb-5350f7b191a8@=pgmasters.net + ControlFile->backupCheckPoint = InvalidXLogRecPtr; ControlFile->backupStartPoint = InvalidXLogRecPtr; + ControlFile->backupStartPointTLI = 0; ControlFile->backupEndPoint = InvalidXLogRecPtr; + ControlFile->backupFromStandby = false; ControlFile->backupEndRequired = false; Still, I get the temptation of being consistent with the current style on HEAD to reset everything, as well.. -- Michael Attachments: [text/x-diff] recovery-in-pgcontrol-v7-0001-add-info-to-manifest.patch (11.0K, ../../[email protected]/2-recovery-in-pgcontrol-v7-0001-add-info-to-manifest.patch) download | inline diff: From 97bb113b5bf5427449b748c3ee25b647a2c5fef5 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Sun, 19 Nov 2023 16:54:36 +0000 Subject: Add label and start/stop time to backup manifest. Add label passed by the user to pg_basebackup and backup start/stop time to the backup_manifest file. Currently these fields are purely for informational purposes. --- doc/src/sgml/backup-manifest.sgml | 51 +++++++++++++++++++++ src/backend/backup/backup_manifest.c | 56 +++++++++++++++++++++++- src/backend/backup/basebackup.c | 9 +++- src/bin/pg_verifybackup/parse_manifest.c | 40 +++++++++++++++++ src/include/backup/backup_manifest.h | 5 ++- 5 files changed, 157 insertions(+), 4 deletions(-) diff --git a/doc/src/sgml/backup-manifest.sgml b/doc/src/sgml/backup-manifest.sgml index 771be1310a..a80b79e587 100644 --- a/doc/src/sgml/backup-manifest.sgml +++ b/doc/src/sgml/backup-manifest.sgml @@ -42,6 +42,16 @@ </listitem> </varlistentry> + <varlistentry> + <term><literal>Backup-Label</literal></term> + <listitem> + <para> + Backup label specified by the user. This will be set to a default value + if no backup label was specified. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>Files</literal></term> <listitem> @@ -67,6 +77,17 @@ </listitem> </varlistentry> + <varlistentry> + <term><literal>Time-Range</literal></term> + <listitem> + <para> + The associated value is always an object describing the start/stop time + of the backup. The structure of this object is further described in + <xref linkend="backup-manifest-time-range" />. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>Manifest-Checksum</literal></term> <listitem> @@ -213,4 +234,34 @@ for the same timeline. </para> </sect1> + + <sect1 id="backup-manifest-time-range"> + <title>Backup Manifest Time Range Object</title> + + <para> + The object which describes the time range always has three keys: + </para> + + <variablelist> + <varlistentry> + <term><literal>Start-Time</literal></term> + <listitem> + <para> + The start time for the backup. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>End-Time</literal></term> + <listitem> + <para> + The end time for the backup. This is when the backup was stopped in + <productname>PostgreSQL</productname> and represents the earliest time + that can be used for time-based Point-In-Time Recovery. + </para> + </listitem> + </varlistentry> + </variablelist> + </sect1> </chapter> diff --git a/src/backend/backup/backup_manifest.c b/src/backend/backup/backup_manifest.c index aeed362a9a..9540f05964 100644 --- a/src/backend/backup/backup_manifest.c +++ b/src/backend/backup/backup_manifest.c @@ -56,7 +56,8 @@ IsManifestEnabled(backup_manifest_info *manifest) void InitializeBackupManifest(backup_manifest_info *manifest, backup_manifest_option want_manifest, - pg_checksum_type manifest_checksum_type) + pg_checksum_type manifest_checksum_type, + const char *label) { memset(manifest, 0, sizeof(backup_manifest_info)); manifest->checksum_type = manifest_checksum_type; @@ -78,9 +79,21 @@ InitializeBackupManifest(backup_manifest_info *manifest, manifest->still_checksumming = true; if (want_manifest != MANIFEST_OPTION_NO) + { + StringInfoData buf; + AppendToManifest(manifest, "{ \"PostgreSQL-Backup-Manifest-Version\": 1,\n" - "\"Files\": ["); + "\"Backup-Label\": "); + + /* JSON encode label and add it to manifest */ + initStringInfo(&buf); + escape_json(&buf, label); + AppendStringToManifest(manifest, buf.data); + pfree(buf.data); + + AppendToManifest(manifest, ",\n\"Files\": ["); + } } /* @@ -308,6 +321,45 @@ AddWALInfoToBackupManifest(backup_manifest_info *manifest, XLogRecPtr startptr, AppendStringToManifest(manifest, "\n],\n"); } +/* + * Add backup start/end time information to the manifest. + */ +void +AddTimeInfoToBackupManifest(backup_manifest_info *manifest, pg_time_t starttime, + pg_time_t endtime) +{ + StringInfoData buf; + + if (!IsManifestEnabled(manifest)) + return; + + /* Start the time range. */ + AppendStringToManifest(manifest, "\"Time-Range\": { "); + + /* + * Convert start/end time to strings and append them to the manifest. Since + * it's not clear what time zone to use and since time zone definitions can + * change, possibly causing confusion, use GMT always. + */ + initStringInfo(&buf); + + appendStringInfoString(&buf, "\"Start-Time\": \""); + enlargeStringInfo(&buf, 128); + buf.len += pg_strftime(&buf.data[buf.len], 128, "%Y-%m-%d %H:%M:%S %Z", + pg_gmtime(&starttime)); + appendStringInfoString(&buf, "\", \"End-Time\": \""); + enlargeStringInfo(&buf, 128); + buf.len += pg_strftime(&buf.data[buf.len], 128, "%Y-%m-%d %H:%M:%S %Z", + pg_gmtime(&endtime)); + appendStringInfoString(&buf, "\" },\n"); + + /* Add to the manifest. */ + AppendStringToManifest(manifest, buf.data); + + /* Avoid leaking memory. */ + pfree(buf.data); +} + /* * Finalize the backup manifest, and send it to the client. */ diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index 35dd79babc..c7b3ba3e6e 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -225,6 +225,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) bbsink_state state; XLogRecPtr endptr; TimeLineID endtli; + pg_time_t starttime; + pg_time_t stoptime; backup_manifest_info manifest; BackupState *backup_state; StringInfo tablespace_map; @@ -243,7 +245,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) backup_started_in_recovery = RecoveryInProgress(); InitializeBackupManifest(&manifest, opt->manifest, - opt->manifest_checksum_type); + opt->manifest_checksum_type, opt->label); total_checksum_failures = 0; @@ -380,6 +382,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) endptr = backup_state->stoppoint; endtli = backup_state->stoptli; + /* Record start/stop time for manifest */ + starttime = backup_state->starttime; + stoptime = backup_state->stoptime; + /* Deallocate backup-related variables. */ pfree(tablespace_map->data); pfree(tablespace_map); @@ -629,6 +635,7 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) AddWALInfoToBackupManifest(&manifest, state.startptr, state.starttli, endptr, endtli); + AddTimeInfoToBackupManifest(&manifest, starttime, stoptime); SendBackupManifest(&manifest, sink); diff --git a/src/bin/pg_verifybackup/parse_manifest.c b/src/bin/pg_verifybackup/parse_manifest.c index bf0227c668..408af88e58 100644 --- a/src/bin/pg_verifybackup/parse_manifest.c +++ b/src/bin/pg_verifybackup/parse_manifest.c @@ -25,6 +25,7 @@ typedef enum JM_EXPECT_TOPLEVEL_END, JM_EXPECT_TOPLEVEL_FIELD, JM_EXPECT_VERSION_VALUE, + JM_EXPECT_BACKUP_LABEL_VALUE, JM_EXPECT_FILES_START, JM_EXPECT_FILES_NEXT, JM_EXPECT_THIS_FILE_FIELD, @@ -33,6 +34,9 @@ typedef enum JM_EXPECT_WAL_RANGES_NEXT, JM_EXPECT_THIS_WAL_RANGE_FIELD, JM_EXPECT_THIS_WAL_RANGE_VALUE, + JM_EXPECT_TIME_RANGE_START, + JM_EXPECT_THIS_TIME_RANGE_FIELD, + JM_EXPECT_THIS_TIME_RANGE_VALUE, JM_EXPECT_MANIFEST_CHECKSUM_VALUE, JM_EXPECT_EOF, } JsonManifestSemanticState; @@ -188,6 +192,9 @@ json_manifest_object_start(void *state) parse->start_lsn = NULL; parse->end_lsn = NULL; break; + case JM_EXPECT_TIME_RANGE_START: + parse->state = JM_EXPECT_THIS_TIME_RANGE_FIELD; + break; default: json_manifest_parse_failure(parse->context, "unexpected object start"); @@ -223,6 +230,9 @@ json_manifest_object_end(void *state) json_manifest_finalize_wal_range(parse); parse->state = JM_EXPECT_WAL_RANGES_NEXT; break; + case JM_EXPECT_THIS_TIME_RANGE_FIELD: + parse->state = JM_EXPECT_TOPLEVEL_FIELD; + break; default: json_manifest_parse_failure(parse->context, "unexpected object end"); @@ -312,6 +322,13 @@ json_manifest_object_field_start(void *state, char *fname, bool isnull) break; } + /* Is this the backup label? */ + if (strcmp(fname, "Backup-Label") == 0) + { + parse->state = JM_EXPECT_BACKUP_LABEL_VALUE; + break; + } + /* Is this the list of files? */ if (strcmp(fname, "Files") == 0) { @@ -326,6 +343,13 @@ json_manifest_object_field_start(void *state, char *fname, bool isnull) break; } + /* Is this the time range? */ + if (strcmp(fname, "Time-Range") == 0) + { + parse->state = JM_EXPECT_TIME_RANGE_START; + break; + } + /* Is this the manifest checksum? */ if (strcmp(fname, "Manifest-Checksum") == 0) { @@ -372,6 +396,14 @@ json_manifest_object_field_start(void *state, char *fname, bool isnull) parse->state = JM_EXPECT_THIS_WAL_RANGE_VALUE; break; + case JM_EXPECT_THIS_TIME_RANGE_FIELD: + if (strcmp(fname, "Start-Time") != 0 && + strcmp(fname, "End-Time") != 0) + json_manifest_parse_failure(parse->context, + "unexpected time range field"); + parse->state = JM_EXPECT_THIS_TIME_RANGE_VALUE; + break; + default: json_manifest_parse_failure(parse->context, "unexpected object field"); @@ -410,6 +442,10 @@ json_manifest_scalar(void *state, char *token, JsonTokenType tokentype) parse->state = JM_EXPECT_TOPLEVEL_FIELD; break; + case JM_EXPECT_BACKUP_LABEL_VALUE: + parse->state = JM_EXPECT_TOPLEVEL_FIELD; + break; + case JM_EXPECT_THIS_FILE_VALUE: switch (parse->file_field) { @@ -451,6 +487,10 @@ json_manifest_scalar(void *state, char *token, JsonTokenType tokentype) parse->state = JM_EXPECT_THIS_WAL_RANGE_FIELD; break; + case JM_EXPECT_THIS_TIME_RANGE_VALUE: + parse->state = JM_EXPECT_THIS_TIME_RANGE_FIELD; + break; + case JM_EXPECT_MANIFEST_CHECKSUM_VALUE: parse->state = JM_EXPECT_TOPLEVEL_END; parse->manifest_checksum = token; diff --git a/src/include/backup/backup_manifest.h b/src/include/backup/backup_manifest.h index bd7067ae42..4ab1291bba 100644 --- a/src/include/backup/backup_manifest.h +++ b/src/include/backup/backup_manifest.h @@ -37,7 +37,8 @@ typedef struct backup_manifest_info extern void InitializeBackupManifest(backup_manifest_info *manifest, backup_manifest_option want_manifest, - pg_checksum_type manifest_checksum_type); + pg_checksum_type manifest_checksum_type, + const char *label); extern void AddFileToBackupManifest(backup_manifest_info *manifest, Oid spcoid, const char *pathname, size_t size, @@ -47,6 +48,8 @@ extern void AddWALInfoToBackupManifest(backup_manifest_info *manifest, XLogRecPtr startptr, TimeLineID starttli, XLogRecPtr endptr, TimeLineID endtli); +extern void AddTimeInfoToBackupManifest(backup_manifest_info *manifest, + pg_time_t starttime, pg_time_t endtime); extern void SendBackupManifest(backup_manifest_info *manifest, bbsink *sink); extern void FreeBackupManifest(backup_manifest_info *manifest); -- 2.34.1 [text/x-diff] recovery-in-pgcontrol-v7-0002-remove-backuplabel.patch (61.1K, ../../[email protected]/3-recovery-in-pgcontrol-v7-0002-remove-backuplabel.patch) download | inline diff: From 6b732c6e085a4ef2ec717af943b44d02cb4b9849 Mon Sep 17 00:00:00 2001 From: David Steele <[email protected]> Date: Sun, 19 Nov 2023 16:54:37 +0000 Subject: Add recovery to pg_control and remove backup_label. Simplify and harden recovery by removing backup_label and storing recovery information directly in pg_control. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If pg_control is removed the cluster will not start. The user may try pg_resetwal, but that tool makes it pretty clear that corruption will result from its use. We could also modify pg_resetwal to complain if recovery info is present in pg_control. * We don't need to worry about backup software seeing a torn copy of pg_control, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_control via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup software to copy pg_control last. In fact the backup software should not copy pg_control from PGDATA at all. Since backup_label is now gone, the fields that used to be in backup_label are now provided as columns returned from pg_backup_start() and pg_backup_stop() and the backup history file is still written to the archive. For pg_basebackup, the label passed on the cmd line as --label is now stored in the manifest. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 31 ++- doc/src/sgml/func.sgml | 41 ++- doc/src/sgml/ref/pg_rewind.sgml | 6 +- src/backend/access/transam/xlog.c | 67 ++--- src/backend/access/transam/xlogbackup.c | 39 ++- src/backend/access/transam/xlogfuncs.c | 50 ++-- src/backend/access/transam/xlogrecovery.c | 249 ++++--------------- src/backend/backup/basebackup.c | 37 +-- src/backend/catalog/system_functions.sql | 10 +- src/backend/utils/misc/pg_controldata.c | 22 +- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 15 +- src/bin/pg_controldata/pg_controldata.c | 8 + src/bin/pg_resetwal/pg_resetwal.c | 4 + src/bin/pg_rewind/filemap.c | 7 +- src/bin/pg_rewind/pg_rewind.c | 64 +---- src/include/access/xlog.h | 2 - src/include/access/xlogbackup.h | 12 +- src/include/access/xlogrecovery.h | 3 +- src/include/catalog/pg_control.h | 19 +- src/include/catalog/pg_proc.dat | 16 +- 20 files changed, 293 insertions(+), 409 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 8cb24d6ae5..584384875b 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -935,19 +935,20 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); ready to archive. </para> <para> - <function>pg_backup_stop</function> will return one row with three - values. The second of these fields should be written to a file named - <filename>backup_label</filename> in the root directory of the backup. The - third field should be written to a file named - <filename>tablespace_map</filename> unless the field is empty. These files are + <function>pg_backup_stop</function> returns the + <filename>pg_control</filename> file, which must be stored in the + <filename>global</filename> directory of the backup. It also returns the + <filename>tablespace_map</filename> file, which should be written in the + root directory of the backup unless the field is empty. These files are vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. </para> </listitem> <listitem> <para> Once the WAL segment files active during the backup are archived, you are - done. The file identified by <function>pg_backup_stop</function>'s first return + done. The file identified by <function>pg_backup_stop</function>'s + <parameter>lsn</parameter> return value is the last segment that is required to form a complete set of backup files. On a primary, if <varname>archive_mode</varname> is enabled and the <literal>wait_for_archive</literal> parameter is <literal>true</literal>, @@ -1013,7 +1014,15 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - You should, however, omit from the backup the files within the + You must exclude <filename>global/pg_control</filename> from your backup + and put the contents of the <parameter>pg_control_file</parameter> column + returned from <function>pg_backup_stop</function> in your backup at + <filename>global/pg_control</filename>. This file contains the information + required to safely recover. + </para> + + <para> + You should also omit from the backup the files within the cluster's <filename>pg_wal/</filename> subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if @@ -1062,11 +1071,11 @@ SELECT * FROM pg_backup_stop(wait_for_archive => true); </para> <para> - The backup label - file includes the label string you gave to <function>pg_backup_start</function>, + The backup history file (which is archived like WAL) includes the label + string you gave to <function>pg_backup_start</function>, as well as the time at which <function>pg_backup_start</function> was run, and the name of the starting WAL file. In case of confusion it is therefore - possible to look inside a backup file and determine exactly which + possible to look inside a backup history file and determine exactly which backup session the dump file came from. The tablespace map file includes the symbolic link names as they exist in the directory <filename>pg_tblspc/</filename> and the full path of each symbolic link. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 93f068edcf..5f27cce161 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26865,7 +26865,10 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <parameter>label</parameter> <type>text</type> <optional>, <parameter>fast</parameter> <type>boolean</type> </optional> ) - <returnvalue>pg_lsn</returnvalue> + <returnvalue>record</returnvalue> + ( <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>start</parameter> <type>timestamptz</type> ) </para> <para> Prepares the server to begin an on-line backup. The only required @@ -26877,6 +26880,13 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 as possible. This forces an immediate checkpoint which will cause a spike in I/O operations, slowing any concurrently executing queries. </para> + <para> + The result columns contain information about the start of the backup + and can be ignored: the <parameter>lsn</parameter> column holds the + starting write-ahead log location, the + <parameter>timeline_id</parameter> column holds the starting timeline, + and the <parameter>stop</parameter> column holds the starting timestamp. + </para> <para> This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function. @@ -26892,13 +26902,15 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 <optional><parameter>wait_for_archive</parameter> <type>boolean</type> </optional> ) <returnvalue>record</returnvalue> - ( <parameter>lsn</parameter> <type>pg_lsn</type>, - <parameter>labelfile</parameter> <type>text</type>, - <parameter>spcmapfile</parameter> <type>text</type> ) + ( <parameter>pg_control_file</parameter> <type>text</type>, + <parameter>tablespace_map_file</parameter> <type>text</type>, + <parameter>lsn</parameter> <type>pg_lsn</type>, + <parameter>timeline_id</parameter> <type>int8</type>, + <parameter>stop</parameter> <type>timestamptz</type> ) </para> <para> Finishes performing an on-line backup. The desired contents of the - backup label file and the tablespace map file are returned as part of + pg_control file and the tablespace map file are returned as part of the result of the function and must be written to files in the backup area. These files must not be written to the live data directory (doing so will cause PostgreSQL to fail to restart in the event of a @@ -26930,13 +26942,18 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 backup. </para> <para> - The result of the function is a single record. - The <parameter>lsn</parameter> column holds the backup's ending - write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These must be - stored as part of the backup and are required as part of the restore - process. + The result of the function is a single record. The first column returns + the contents of the <filename>pg_control</filename> file and the + second column returns the contents of the + <filename>tablespace_map</filename> file. These must be stored as part + of the backup and are required as part of the restore process. Note that + <filename>pg_control</filename> will be 8192KiB to match the on-disk + size of <filename>pg_control</filename>. The remainder of the columns + contain information about the end of the backup and can be ignored: the + <parameter>lsn</parameter> column holds the ending write-ahead log + location, the <parameter>timeline_id</parameter> column holds the ending + timeline, and the <parameter>stop</parameter> column holds the ending + timestamp. </para> <para> This function is restricted to superusers by default, but other users diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 8e0000d39f..404ce7c65e 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -400,7 +400,6 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b <filename>pg_serial/</filename>, <filename>pg_snapshots/</filename>, <filename>pg_stat_tmp/</filename>, and <filename>pg_subtrans/</filename> are omitted from the data copied from the source cluster. The files - <filename>backup_label</filename>, <filename>tablespace_map</filename>, <filename>pg_internal.init</filename>, <filename>postmaster.opts</filename>, and @@ -410,9 +409,8 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, b </step> <step> <para> - Create a <filename>backup_label</filename> file to begin WAL replay at - the checkpoint created at failover and configure the - <filename>pg_control</filename> file with a minimum consistency LSN + Update <filename>pg_control</filename> file to begin WAL replay at + the checkpoint created at failover and with a minimum consistency LSN defined as the result of <literal>pg_current_wal_insert_lsn()</literal> when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 1159dff1a6..928cad0651 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -74,6 +74,7 @@ #include "pg_trace.h" #include "pgstat.h" #include "port/atomics.h" +#include "port/pg_crc32c.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" #include "postmaster/startup.h" @@ -5135,7 +5136,6 @@ StartupXLOG(void) bool wasShutdown; bool didCrash; bool haveTblspcMap; - bool haveBackupLabel; XLogRecPtr EndOfLog; TimeLineID EndOfLogTLI; TimeLineID newTLI; @@ -5259,13 +5259,14 @@ StartupXLOG(void) /* * Prepare for WAL recovery if needed. * - * InitWalRecovery analyzes the control file and the backup label file, if - * any. It updates the in-memory ControlFile buffer according to the - * starting checkpoint, and sets InRecovery and ArchiveRecoveryRequested. + * InitWalRecovery analyzes the control file and checks if backup recovery + * has been requested. It updates the in-memory ControlFile buffer + * according to the starting checkpoint, and sets InRecovery and + * ArchiveRecoveryRequested. + * * It also applies the tablespace map file, if any. */ - InitWalRecovery(ControlFile, &wasShutdown, - &haveBackupLabel, &haveTblspcMap); + InitWalRecovery(ControlFile, &wasShutdown, &haveTblspcMap); checkPoint = ControlFile->checkPointCopy; /* initialize shared memory variables from the checkpoint record */ @@ -5408,20 +5409,6 @@ StartupXLOG(void) */ UpdateControlFile(); - /* - * If there was a backup label file, it's done its job and the info - * has now been propagated into pg_control. We must get rid of the - * label file so that if we crash during recovery, we'll pick up at - * the latest recovery restartpoint instead of going all the way back - * to the backup start point. It seems prudent though to just rename - * the file out of the way rather than delete it completely. - */ - if (haveBackupLabel) - { - unlink(BACKUP_LABEL_OLD); - durable_rename(BACKUP_LABEL_FILE, BACKUP_LABEL_OLD, FATAL); - } - /* * If there was a tablespace_map file, it's done its job and the * symlinks have been created. We must get rid of the map file so @@ -5571,10 +5558,8 @@ StartupXLOG(void) * (at which point we reset backupStartPoint to be Invalid), for * backup-from-replica (which can't inject records into the WAL stream), * that point is when we reach the minRecoveryPoint in pg_control (which - * we purposefully copy last when backing up from a replica). For - * pg_rewind (which creates a backup_label with a method of "pg_rewind") - * or snapshot-style backups (which don't), backupEndRequired will be set - * to false. + * we purposefully copy last when backing up). For pg_rewind or + * snapshot-style backups, backupEndRequired will be set to false. * * Note: it is indeed okay to look at the local variable * LocalMinRecoveryPoint here, even though ControlFile->minRecoveryPoint @@ -5951,8 +5936,11 @@ ReachedEndOfBackup(XLogRecPtr EndRecPtr, TimeLineID tli) ControlFile->minRecoveryPointTLI = tli; } + ControlFile->backupCheckPoint = InvalidXLogRecPtr; ControlFile->backupStartPoint = InvalidXLogRecPtr; + ControlFile->backupStartPointTLI = 0; ControlFile->backupEndPoint = InvalidXLogRecPtr; + ControlFile->backupFromStandby = false; ControlFile->backupEndRequired = false; UpdateControlFile(); @@ -8744,11 +8732,33 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) int seconds_before_warning; int waits = 0; bool reported_waiting = false; + ControlFileData *controlFileCopy = (ControlFileData *)state->controlFile; Assert(state != NULL); backup_stopped_in_recovery = RecoveryInProgress(); + /* + * Create a copy of control data and update it with fields required for + * recovery. Also recalculate the CRC. + */ + memset(controlFileCopy, 0, PG_CONTROL_FILE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupRecoveryRequired = true; + controlFileCopy->backupFromStandby = backup_stopped_in_recovery; + controlFileCopy->backupEndRequired = true; + controlFileCopy->backupCheckPoint = state->checkpointloc; + controlFileCopy->backupStartPoint = state->startpoint; + controlFileCopy->backupStartPointTLI = state->starttli; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileData, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. @@ -8850,11 +8860,8 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) "Enable full_page_writes and run CHECKPOINT on the primary, " "and then try an online backup again."))); - - LWLockAcquire(ControlFileLock, LW_SHARED); - state->stoppoint = ControlFile->minRecoveryPoint; - state->stoptli = ControlFile->minRecoveryPointTLI; - LWLockRelease(ControlFileLock); + state->stoppoint = controlFileCopy->minRecoveryPoint; + state->stoptli = controlFileCopy->minRecoveryPointTLI; } else { @@ -8896,7 +8903,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive) histfilepath))); /* Build and save the contents of the backup history file */ - history_file = build_backup_content(state, true); + history_file = build_backup_history_content(state); fprintf(fp, "%s", history_file); pfree(history_file); diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c index 21d68133ae..22c95f3c4c 100644 --- a/src/backend/access/transam/xlogbackup.c +++ b/src/backend/access/transam/xlogbackup.c @@ -18,19 +18,19 @@ #include "access/xlogbackup.h" /* - * Build contents for backup_label or backup history file. - * - * When ishistoryfile is true, it creates the contents for a backup history - * file, otherwise it creates contents for a backup_label file. + * Build contents for backup history file. * * Returns the result generated as a palloc'd string. */ char * -build_backup_content(BackupState *state, bool ishistoryfile) +build_backup_history_content(BackupState *state) { char startstrbuf[128]; + char stopstrfbuf[128]; char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */ + char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ XLogSegNo startsegno; + XLogSegNo stopsegno; StringInfo result = makeStringInfo(); char *data; @@ -45,16 +45,10 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n", LSN_FORMAT_ARGS(state->startpoint), startxlogfile); - if (ishistoryfile) - { - char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ - XLogSegNo stopsegno; - - XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); - XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); - appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", - LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); - } + XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); + XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); + appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", + LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n", LSN_FORMAT_ARGS(state->checkpointloc)); @@ -65,17 +59,12 @@ build_backup_content(BackupState *state, bool ishistoryfile) appendStringInfo(result, "LABEL: %s\n", state->name); appendStringInfo(result, "START TIMELINE: %u\n", state->starttli); - if (ishistoryfile) - { - char stopstrfbuf[128]; - - /* Use the log timezone here, not the session timezone */ - pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", - pg_localtime(&state->stoptime, log_timezone)); + /* Use the log timezone here, not the session timezone */ + pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", + pg_localtime(&state->stoptime, log_timezone)); - appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); - appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); - } + appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); + appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); data = result->data; pfree(result); diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 45a70668b1..d81d36705f 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -53,7 +53,7 @@ static MemoryContext backupcontext = NULL; * pg_backup_start: set up for taking an on-line backup dump * * Essentially what this does is to create the contents required for the - * backup_label file and the tablespace map. + * the tablespace map. * * Permission checking for this function is managed through the normal * GRANT system. @@ -61,6 +61,10 @@ static MemoryContext backupcontext = NULL; Datum pg_backup_start(PG_FUNCTION_ARGS) { +#define PG_BACKUP_START_V2_COLS 3 + TupleDesc tupdesc; + Datum values[PG_BACKUP_START_V2_COLS] = {0}; + bool nulls[PG_BACKUP_START_V2_COLS] = {0}; text *backupid = PG_GETARG_TEXT_PP(0); bool fast = PG_GETARG_BOOL(1); char *backupidstr; @@ -69,6 +73,10 @@ pg_backup_start(PG_FUNCTION_ARGS) backupidstr = text_to_cstring(backupid); + /* Initialize attributes information in the tuple descriptor */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + if (status == SESSION_BACKUP_RUNNING) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -102,7 +110,12 @@ pg_backup_start(PG_FUNCTION_ARGS) register_persistent_abort_backup_handler(); do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map); - PG_RETURN_LSN(backup_state->startpoint); + values[0] = LSNGetDatum(backup_state->startpoint); + values[1] = Int64GetDatum(backup_state->starttli); + values[2] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->starttime)); + + /* Returns the record as Datum */ + PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); } @@ -113,14 +126,12 @@ pg_backup_start(PG_FUNCTION_ARGS) * allows the user to choose if they want to wait for the WAL to be archived * or if we should just return as soon as the WAL record is written. * - * This function stops an in-progress backup, creates backup_label contents and - * it returns the backup stop LSN, backup_label and tablespace_map contents. + * This function stops an in-progress backup and returns the backup stop LSN, + * pg_control and tablespace_map contents. * - * The backup_label contains the user-supplied label string (typically this - * would be used to tell where the backup dump will be stored), the starting - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * The pg_control file contains the recovery information for the backup. It is + * the caller's responsibility to write the pg_control and tablespace_map files + * in the data folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -128,12 +139,12 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 5 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] = {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] = {0}; bool waitforarchive = PG_GETARG_BOOL(0); - char *backup_label; + bytea *pg_control_bytea; SessionBackupState status = get_backup_status(); /* Initialize attributes information in the tuple descriptor */ @@ -152,15 +163,16 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); - /* Build the contents of backup_label */ - backup_label = build_backup_content(backup_state, false); - - values[0] = LSNGetDatum(backup_state->stoppoint); - values[1] = CStringGetTextDatum(backup_label); - values[2] = CStringGetTextDatum(tablespace_map->data); + /* Build the contents of pg_control */ + pg_control_bytea = (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_FILE_SIZE); - /* Deallocate backup-related variables */ - pfree(backup_label); + values[0] = PointerGetDatum(pg_control_bytea); + values[1] = CStringGetTextDatum(tablespace_map->data); + values[2] = LSNGetDatum(backup_state->stoppoint); + values[3] = Int64GetDatum(backup_state->stoptli); + values[4] = TimestampTzGetDatum(time_t_to_timestamptz(backup_state->stoptime)); /* Clean up the session-level state and its memory context */ backup_state = NULL; diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index c61566666a..d05ff41f60 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -6,7 +6,7 @@ * This source file contains functions controlling WAL recovery. * InitWalRecovery() initializes the system for crash or archive recovery, * or standby mode, depending on configuration options and the state of - * the control file and possible backup label file. PerformWalRecovery() + * the control file and possible backup recovery. PerformWalRecovery() * performs the actual WAL replay, calling the rmgr-specific redo routines. * FinishWalRecovery() performs end-of-recovery checks and cleanup actions, * and prepares information needed to initialize the WAL for writes. In @@ -152,11 +152,12 @@ static bool recovery_signal_file_found = false; /* * CheckPointLoc is the position of the checkpoint record that determines - * where to start the replay. It comes from the backup label file or the - * control file. + * where to start the replay. It comes from the control file, either from the + * default location or from a backup recovery field. * - * RedoStartLSN is the checkpoint's REDO location, also from the backup label - * file or the control file. In standby mode, XLOG streaming usually starts + * RedoStartLSN is the checkpoint's REDO location, also from the default + * control file location or from a backup recovery field. In standby mode, + * XLOG streaming usually starts * from the position where an invalid record was found. But if we fail to * read even the initial checkpoint record, we use the REDO location instead * of the checkpoint location as the start position of XLOG streaming. @@ -388,9 +389,6 @@ static void ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, Time static void EnableStandbyMode(void); static void readRecoverySignalFile(void); static void validateRecoveryParameters(void); -static bool read_backup_label(XLogRecPtr *checkPointLoc, - TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby); static bool read_tablespace_map(List **tablespaces); static void xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTLI); @@ -492,8 +490,8 @@ EnableStandbyMode(void) * Prepare the system for WAL recovery, if needed. * * This is called by StartupXLOG() which coordinates the server startup - * sequence. This function analyzes the control file and the backup label - * file, if any, and figures out whether we need to perform crash recovery or + * sequence. This function analyzes the control file and backup recovery + * info, if any, and figures out whether we need to perform crash recovery or * archive recovery, and how far we need to replay the WAL to reach a * consistent state. * @@ -510,7 +508,7 @@ EnableStandbyMode(void) */ void InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, - bool *haveBackupLabel_ptr, bool *haveTblspcMap_ptr) + bool *haveTblspcMap_ptr) { XLogPageReadPrivate *private; struct stat st; @@ -518,7 +516,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, XLogRecord *record; DBState dbstate_at_startup; bool haveTblspcMap = false; - bool haveBackupLabel = false; + bool backupRecoveryRequired = false; CheckPoint checkPoint; bool backupFromStandby = false; @@ -549,7 +547,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Set the WAL reading processor now, as it will be needed when reading - * the checkpoint record required (backup_label or not). + * the checkpoint record required (backup recovery required or not). */ private = palloc0(sizeof(XLogPageReadPrivate)); xlogreader = @@ -585,18 +583,33 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, primary_image_masked = (char *) palloc(BLCKSZ); /* - * Read the backup_label file. We want to run this part of the recovery - * process after checking for signal files and after performing validation - * of the recovery parameters. + * Load recovery settings from pg_control. We want to run this part of the + * recovery process after checking for signal files and after performing + * validation of the recovery parameters. */ - if (read_backup_label(&CheckPointLoc, &CheckPointTLI, &backupEndRequired, - &backupFromStandby)) + if (ControlFile->backupRecoveryRequired) { List *tablespaces = NIL; + /* Initialize recovery from fields stored in pg_control */ + CheckPointLoc = ControlFile->backupCheckPoint; + CheckPointTLI = ControlFile->backupStartPointTLI; + RedoStartLSN = ControlFile->backupStartPoint; + RedoStartTLI = ControlFile->backupStartPointTLI; + backupEndRequired = ControlFile->backupEndRequired; + backupFromStandby = ControlFile->backupFromStandby; + + /* + * Clear backupRecoveryRequired in ControlFile so we do not initialize + * recovery settings again, but also set a local variable so later logic + * knows that backup recovery was initialized. + */ + ControlFile->backupRecoveryRequired = false; + backupRecoveryRequired = true; + /* - * Archive recovery was requested, and thanks to the backup label - * file, we know how far we need to replay to reach consistency. Enter + * Archive recovery was requested, and thanks to the recovery + * info, we know how far we need to replay to reach consistency. Enter * archive recovery directly. */ InArchiveRecovery = true; @@ -604,8 +617,9 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, EnableStandbyMode(); /* - * When a backup_label file is present, we want to roll forward from - * the checkpoint it identifies, rather than using pg_control. + * When backup recovery is requested, we want to roll forward from + * the checkpoint it identifies, rather than using the default + * checkpoint. */ record = ReadCheckpointRecord(xlogprefetcher, CheckPointLoc, CheckPointTLI); @@ -620,9 +634,8 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* * Make sure that REDO location exists. This may not be the case - * if there was a crash during an online backup, which left a - * backup_label around that references a WAL segment that's - * already been archived. + * if recovery.signal is missing and the WAL has already been + * archived. */ if (checkPoint.redo < CheckPointLoc) { @@ -631,20 +644,16 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, checkPoint.ThisTimeLineID)) ereport(FATAL, (errmsg("could not find redo location referenced by checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); } } else { ereport(FATAL, (errmsg("could not locate required checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.signal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); wasShutdown = false; /* keep compiler quiet */ } @@ -679,37 +688,32 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, /* tell the caller to delete it later */ haveTblspcMap = true; } - - /* tell the caller to delete it later */ - haveBackupLabel = true; } else { - /* No backup_label file has been found if we are here. */ - /* - * If tablespace_map file is present without backup_label file, there - * is no use of such file. There is no harm in retaining it, but it - * is better to get rid of the map file so that we don't have any + * If tablespace_map file is present without backup recovery requested, + * there is no use of such file. There is no harm in retaining it, but + * it is better to get rid of the map file so that we don't have any * redundant file in data directory and it will avoid any sort of * confusion. It seems prudent though to just rename the file out of * the way rather than delete it completely, also we ignore any error * that occurs in rename operation as even if map file is present - * without backup_label file, it is harmless. + * without backup recovery requested, it is harmless. */ if (stat(TABLESPACE_MAP, &st) == 0) { unlink(TABLESPACE_MAP_OLD); if (durable_rename(TABLESPACE_MAP, TABLESPACE_MAP_OLD, DEBUG1) == 0) ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("File \"%s\" was renamed to \"%s\".", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); else ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not requested", + TABLESPACE_MAP), errdetail("Could not rename file \"%s\" to \"%s\": %m.", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); } @@ -943,7 +947,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. */ - if (haveBackupLabel) + if (backupRecoveryRequired) { ControlFile->backupStartPoint = checkPoint.redo; ControlFile->backupEndRequired = backupEndRequired; @@ -953,7 +957,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, if (dbstate_at_startup != DB_IN_ARCHIVE_RECOVERY && dbstate_at_startup != DB_SHUTDOWNED_IN_RECOVERY) ereport(FATAL, - (errmsg("backup_label contains data inconsistent with control file"), + (errmsg("pg_control contains inconsistent data for standby backup"), errhint("This means that the backup is corrupted and you will " "have to use another backup for recovery."))); ControlFile->backupEndPoint = ControlFile->minRecoveryPoint; @@ -983,7 +987,6 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, missingContrecPtr = InvalidXLogRecPtr; *wasShutdown_ptr = wasShutdown; - *haveBackupLabel_ptr = haveBackupLabel; *haveTblspcMap_ptr = haveTblspcMap; } @@ -1156,154 +1159,6 @@ validateRecoveryParameters(void) } } -/* - * read_backup_label: check to see if a backup_label file is present - * - * If we see a backup_label during recovery, we assume that we are recovering - * from a backup dump file, and we therefore roll forward from the checkpoint - * identified by the label file, NOT what pg_control says. This avoids the - * problem that pg_control might have been archived one or more checkpoints - * later than the start of the dump, and so if we rely on it as the start - * point, we will fail to restore a consistent database state. - * - * Returns true if a backup_label was found (and fills the checkpoint - * location and TLI into *checkPointLoc and *backupLabelTLI, respectively); - * returns false if not. If this backup_label came from a streamed backup, - * *backupEndRequired is set to true. If this backup_label was created during - * recovery, *backupFromStandby is set to true. - * - * Also sets the global variables RedoStartLSN and RedoStartTLI with the LSN - * and TLI read from the backup file. - */ -static bool -read_backup_label(XLogRecPtr *checkPointLoc, TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby) -{ - char startxlogfilename[MAXFNAMELEN]; - TimeLineID tli_from_walseg, - tli_from_file; - FILE *lfp; - char ch; - char backuptype[20]; - char backupfrom[20]; - char backuplabel[MAXPGPATH]; - char backuptime[128]; - uint32 hi, - lo; - - /* suppress possible uninitialized-variable warnings */ - *checkPointLoc = InvalidXLogRecPtr; - *backupLabelTLI = 0; - *backupEndRequired = false; - *backupFromStandby = false; - - /* - * See if label file is present - */ - lfp = AllocateFile(BACKUP_LABEL_FILE, "r"); - if (!lfp) - { - if (errno != ENOENT) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - return false; /* it's not there, all is fine */ - } - - /* - * Read and parse the START WAL LOCATION and CHECKPOINT lines (this code - * is pretty crude, but we are not expecting any variability in the file - * format). - */ - if (fscanf(lfp, "START WAL LOCATION: %X/%X (file %08X%16s)%c", - &hi, &lo, &tli_from_walseg, startxlogfilename, &ch) != 5 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - RedoStartLSN = ((uint64) hi) << 32 | lo; - RedoStartTLI = tli_from_walseg; - if (fscanf(lfp, "CHECKPOINT LOCATION: %X/%X%c", - &hi, &lo, &ch) != 3 || ch != '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - *checkPointLoc = ((uint64) hi) << 32 | lo; - *backupLabelTLI = tli_from_walseg; - - /* - * BACKUP METHOD lets us know if this was a typical backup ("streamed", - * which could mean either pg_basebackup or the pg_backup_start/stop - * method was used) or if this label came from somewhere else (the only - * other option today being from pg_rewind). If this was a streamed - * backup then we know that we need to play through until we get to the - * end of the WAL which was generated during the backup (at which point we - * will have reached consistency and backupEndRequired will be reset to be - * false). - */ - if (fscanf(lfp, "BACKUP METHOD: %19s\n", backuptype) == 1) - { - if (strcmp(backuptype, "streamed") == 0) - *backupEndRequired = true; - } - - /* - * BACKUP FROM lets us know if this was from a primary or a standby. If - * it was from a standby, we'll double-check that the control file state - * matches that of a standby. - */ - if (fscanf(lfp, "BACKUP FROM: %19s\n", backupfrom) == 1) - { - if (strcmp(backupfrom, "standby") == 0) - *backupFromStandby = true; - } - - /* - * Parse START TIME and LABEL. Those are not mandatory fields for recovery - * but checking for their presence is useful for debugging and the next - * sanity checks. Cope also with the fact that the result buffers have a - * pre-allocated size, hence if the backup_label file has been generated - * with strings longer than the maximum assumed here an incorrect parsing - * happens. That's fine as only minor consistency checks are done - * afterwards. - */ - if (fscanf(lfp, "START TIME: %127[^\n]\n", backuptime) == 1) - ereport(DEBUG1, - (errmsg_internal("backup time %s in file \"%s\"", - backuptime, BACKUP_LABEL_FILE))); - - if (fscanf(lfp, "LABEL: %1023[^\n]\n", backuplabel) == 1) - ereport(DEBUG1, - (errmsg_internal("backup label %s in file \"%s\"", - backuplabel, BACKUP_LABEL_FILE))); - - /* - * START TIMELINE is new as of 11. Its parsing is not mandatory, still use - * it as a sanity check if present. - */ - if (fscanf(lfp, "START TIMELINE: %u\n", &tli_from_file) == 1) - { - if (tli_from_walseg != tli_from_file) - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE), - errdetail("Timeline ID parsed is %u, but expected %u.", - tli_from_file, tli_from_walseg))); - - ereport(DEBUG1, - (errmsg_internal("backup timeline %u in file \"%s\"", - tli_from_file, BACKUP_LABEL_FILE))); - } - - if (ferror(lfp) || FreeFile(lfp)) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - - return true; -} - /* * read_tablespace_map: check to see if a tablespace_map file is present * diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c index c7b3ba3e6e..f6cd7d954c 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -22,6 +22,7 @@ #include "backup/basebackup.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "commands/defrem.h" #include "common/compression.h" #include "common/file_perm.h" @@ -192,10 +193,9 @@ static const struct exclude_list_item excludeFiles[] = {RELCACHE_INIT_FILENAME, true}, /* - * backup_label and tablespace_map should not exist in a running cluster - * capable of doing an online backup, but exclude them just in case. + * tablespace_map should not exist in a running cluster capable of doing + * an online backup, but exclude it just in case. */ - {BACKUP_LABEL_FILE, false}, {TABLESPACE_MAP, false}, /* @@ -310,19 +310,11 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) if (ti->path == NULL) { - struct stat statbuf; bool sendtblspclinks = true; - char *backup_label; bbsink_begin_archive(sink, "base.tar"); - /* In the main tar, include the backup_label first... */ - backup_label = build_backup_content(backup_state, false); - sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, -1, &manifest); - pfree(backup_label); - - /* Then the tablespace_map file, if required... */ + /* Send the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, @@ -334,15 +326,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid); - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) != 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest); + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + + /* Send copy of pg_control containing recovery info */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_FILE_SIZE, &manifest); } else { @@ -376,9 +367,6 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) } } - basebackup_progress_wait_wal_archive(&state); - do_pg_backup_stop(backup_state, !opt->nowait); - endptr = backup_state->stoppoint; endtli = backup_state->stoptli; @@ -1051,7 +1039,6 @@ sendFileWithContent(bbsink *sink, const char *filename, const char *content, /* * Construct a stat struct for the file we're injecting in the tar. */ - /* Windows doesn't have the concept of uid and gid */ #ifdef WIN32 statbuf.st_uid = 0; diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index 4206752881..1f37719cf2 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -384,13 +384,15 @@ BEGIN ATOMIC END; CREATE OR REPLACE FUNCTION - pg_backup_start(label text, fast boolean DEFAULT false) - RETURNS pg_lsn STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' + pg_backup_start(label text, fast boolean DEFAULT false, OUT lsn pg_lsn, + OUT timeline_id int8, OUT start timestamptz) + RETURNS record STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' PARALLEL RESTRICTED; CREATE OR REPLACE FUNCTION pg_backup_stop ( - wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + wait_for_archive boolean DEFAULT true, OUT pg_control_file bytea, + OUT tablespace_map_file text, OUT lsn pg_lsn, OUT timeline_id int8, + OUT stop timestamptz) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index a1003a464d..44c4e987ce 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -163,8 +163,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[9]; + bool nulls[9]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -187,15 +187,27 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[1] = Int32GetDatum(ControlFile->minRecoveryPointTLI); nulls[1] = false; - values[2] = LSNGetDatum(ControlFile->backupStartPoint); + values[2] = LSNGetDatum(ControlFile->backupCheckPoint); nulls[2] = false; - values[3] = LSNGetDatum(ControlFile->backupEndPoint); + values[3] = LSNGetDatum(ControlFile->backupStartPoint); nulls[3] = false; - values[4] = BoolGetDatum(ControlFile->backupEndRequired); + values[4] = Int32GetDatum(ControlFile->backupStartPointTLI); nulls[4] = false; + values[5] = LSNGetDatum(ControlFile->backupEndPoint); + nulls[5] = false; + + values[6] = BoolGetDatum(ControlFile->backupRecoveryRequired); + nulls[6] = false; + + values[7] = BoolGetDatum(ControlFile->backupFromStandby); + nulls[7] = false; + + values[8] = BoolGetDatum(ControlFile->backupEndRequired); + nulls[8] = false; + htup = heap_form_tuple(tupdesc, values, nulls); PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_basebackup/t/010_pg_basebackup.pl index b9f5e1266b..c655cb0335 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -171,8 +171,8 @@ SKIP: # Write some files to test that they are not copied. foreach my $filename ( - qw(backup_label tablespace_map postgresql.auto.conf.tmp - current_logfiles.tmp global/pg_internal.init.123)) + qw(tablespace_map postgresql.auto.conf.tmp current_logfiles.tmp + global/pg_internal.init.123)) { open my $file, '>>', "$pgdata/$filename"; print $file "DONOTCOPY"; @@ -261,14 +261,13 @@ foreach my $filename (@tempRelationFiles) "base/$postgresOid/$filename not copied"); } -# Make sure existing backup_label was ignored. -isnt(slurp_file("$tempdir/backup/backup_label"), - 'DONOTCOPY', 'existing backup_label not copied'); +# Make sure existing tablespace_map was ignored. +ok(!-f "$tempdir/backup/tablespace_map", 'tablespace_map not in backup'); rmtree("$tempdir/backup"); -# Now delete the bogus backup_label file since it will interfere with startup -unlink("$pgdata/backup_label") - or BAIL_OUT("unable to unlink $pgdata/backup_label"); +# Now delete the bogus tablespace_map file since it will interfere with startup +unlink("$pgdata/tablespace_map") + or BAIL_OUT("unable to unlink $pgdata/tablespace_map"); $node->command_ok( [ diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 93e0837947..cc515b622f 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -277,10 +277,18 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->minRecoveryPoint)); printf(_("Min recovery ending loc's timeline: %u\n"), ControlFile->minRecoveryPointTLI); + printf(_("Backup checkpoint location: %X/%X\n"), + LSN_FORMAT_ARGS(ControlFile->backupCheckPoint)); printf(_("Backup start location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupStartPoint)); + printf(_("Backup start location's timeline: %u\n"), + ControlFile->backupStartPointTLI); printf(_("Backup end location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); + printf(_("Backup recovery required: %s\n"), + ControlFile->backupRecoveryRequired ? _("yes") : _("no")); + printf(_("Backup from standby: %s\n"), + ControlFile->backupFromStandby ? _("yes") : _("no")); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 3ae3fc06df..255101ff3a 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -870,8 +870,12 @@ RewriteControlFile(void) ControlFile.checkPoint = ControlFile.checkPointCopy.redo; ControlFile.minRecoveryPoint = 0; ControlFile.minRecoveryPointTLI = 0; + ControlFile.backupCheckPoint = 0; ControlFile.backupStartPoint = 0; + ControlFile.backupStartPointTLI = 0; ControlFile.backupEndPoint = 0; + ControlFile.backupRecoveryRequired = false; + ControlFile.backupFromStandby = false; ControlFile.backupEndRequired = false; /* diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index ecadd69dc5..213f4e71b8 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -139,11 +139,10 @@ static const struct exclude_list_item excludeFiles[] = {"pg_internal.init", true}, /* defined as RELCACHE_INIT_FILENAME */ /* - * If there is a backup_label or tablespace_map file, it indicates that a - * recovery failed and this cluster probably can't be rewound, but exclude - * them anyway if they are found. + * If there is a tablespace_map file, it indicates that a recovery failed + * and this cluster probably can't be rewound, but exclude it anyway if it + * is found. */ - {"backup_label", false}, /* defined as BACKUP_LABEL_FILE */ {"tablespace_map", false}, /* defined as TABLESPACE_MAP */ /* diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index bfd44a284e..6b23ed4a1c 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -39,9 +39,6 @@ static void perform_rewind(filemap_t *filemap, rewind_source *source, TimeLineID chkpttli, XLogRecPtr chkptredo); -static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, - XLogRecPtr checkpointloc); - static void digestControlFile(ControlFileData *ControlFile, const char *content, size_t size); static void getRestoreCommand(const char *argv0); @@ -651,10 +648,10 @@ perform_rewind(filemap_t *filemap, rewind_source *source, } if (showprogress) - pg_log_info("creating backup label and updating control file"); + pg_log_info("updating control file"); /* - * Create a backup label file, to tell the target where to begin the WAL + * Get recovery fields to tell the target where to begin the WAL * replay. Normally, from the last common checkpoint between the source * and the target. But if the source is a standby server, it's possible * that the last common checkpoint is *after* the standby's restartpoint. @@ -672,7 +669,6 @@ perform_rewind(filemap_t *filemap, rewind_source *source, chkpttli = ControlFile_source.checkPointCopy.ThisTimeLineID; chkptrec = ControlFile_source.checkPoint; } - createBackupLabel(chkptredo, chkpttli, chkptrec); /* * Update control file of target, to tell the target how far it must @@ -722,6 +718,12 @@ perform_rewind(filemap_t *filemap, rewind_source *source, ControlFile_new.minRecoveryPoint = endrec; ControlFile_new.minRecoveryPointTLI = endtli; ControlFile_new.state = DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupRecoveryRequired = true; + ControlFile_new.backupFromStandby = true; + ControlFile_new.backupEndRequired = false; + ControlFile_new.backupCheckPoint = chkptrec; + ControlFile_new.backupStartPoint = chkptredo; + ControlFile_new.backupStartPointTLI = chkpttli; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } @@ -729,7 +731,10 @@ perform_rewind(filemap_t *filemap, rewind_source *source, static void sanityChecks(void) { - /* TODO Check that there's no backup_label in either cluster */ + /* + * TODO Check that neither cluster has backupRecoveryRequested set in + * pg_control. + */ /* Check system_identifier match */ if (ControlFile_target.system_identifier != ControlFile_source.system_identifier) @@ -951,51 +956,6 @@ findCommonAncestorTimeline(TimeLineHistoryEntry *a_history, int a_nentries, } } - -/* - * Create a backup_label file that forces recovery to begin at the last common - * checkpoint. - */ -static void -createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, XLogRecPtr checkpointloc) -{ - XLogSegNo startsegno; - time_t stamp_time; - char strfbuf[128]; - char xlogfilename[MAXFNAMELEN]; - struct tm *tmp; - char buf[1000]; - int len; - - XLByteToSeg(startpoint, startsegno, WalSegSz); - XLogFileName(xlogfilename, starttli, startsegno, WalSegSz); - - /* - * Construct backup label file - */ - stamp_time = time(NULL); - tmp = localtime(&stamp_time); - strftime(strfbuf, sizeof(strfbuf), "%Y-%m-%d %H:%M:%S %Z", tmp); - - len = snprintf(buf, sizeof(buf), - "START WAL LOCATION: %X/%X (file %s)\n" - "CHECKPOINT LOCATION: %X/%X\n" - "BACKUP METHOD: pg_rewind\n" - "BACKUP FROM: standby\n" - "START TIME: %s\n", - /* omit LABEL: line */ - LSN_FORMAT_ARGS(startpoint), xlogfilename, - LSN_FORMAT_ARGS(checkpointloc), - strfbuf); - if (len >= sizeof(buf)) - pg_fatal("backup label buffer too small"); /* shouldn't happen */ - - /* TODO: move old file out of the way, if any. */ - open_target_file("backup_label", true); /* BACKUP_LABEL_FILE */ - write_target_range(buf, 0, len); - close_target_file(); -} - /* * Check CRC of control file */ diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index a14126d164..3aac6839a7 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -293,8 +293,6 @@ extern SessionBackupState get_backup_status(void); /* File path names (all relative to $PGDATA) */ #define RECOVERY_SIGNAL_FILE "recovery.signal" #define STANDBY_SIGNAL_FILE "standby.signal" -#define BACKUP_LABEL_FILE "backup_label" -#define BACKUP_LABEL_OLD "backup_label.old" #define TABLESPACE_MAP "tablespace_map" #define TABLESPACE_MAP_OLD "tablespace_map.old" diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h index 1611358137..9be80a4f7d 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" /* Structure to hold backup state. */ @@ -33,9 +34,16 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. Fields have been + * updated for recovery and the CRC has been recalculated. Bytes after + * sizeof(ControlFileData) are zeroed. + */ + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; } BackupState; -extern char *build_backup_content(BackupState *state, - bool ishistoryfile); +extern char *build_backup_history_content(BackupState *state); #endif /* XLOG_BACKUP_H */ diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrecovery.h index ee0bc74278..981266f734 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -80,8 +80,7 @@ extern Size XLogRecoveryShmemSize(void); extern void XLogRecoveryShmemInit(void); extern void InitWalRecovery(ControlFileData *ControlFile, - bool *wasShutdown_ptr, bool *haveBackupLabel_ptr, - bool *haveTblspcMap_ptr); + bool *wasShutdown_ptr, bool *haveTblspcMap_ptr); extern void PerformWalRecovery(void); /* diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index 2ae72e3b26..0ea2e73368 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -146,6 +146,9 @@ typedef struct ControlFileData * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * + * backupCheckPoint is the backup start checkpoint and is set to zero after + * recovery is initialized. + * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and @@ -160,14 +163,28 @@ typedef struct ControlFileData * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * + * backupRecoveryRequired indicates that the pg_control file was provided + * by a backup or pg_rewind and recovery settings need to be copied to the + * appropriate fields. It will be set to false when the settings have been + * copied. + * + * backupFromStandby indicates that the backup was taken on a standby. It is + * required to initialize recovery and set to false afterwards. + * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely - * start up. + * start up. Currently backupEndRequired should only be false if recovery + * settings were configured by pg_rewind, which does not require an end + * point. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; + XLogRecPtr backupCheckPoint; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; + TimeLineID backupStartPointTLI; + bool backupRecoveryRequired; + bool backupFromStandby; bool backupEndRequired; /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index fb58dee3bc..16346f0540 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6427,13 +6427,17 @@ prosrc => 'pg_terminate_backend' }, { oid => '2172', descr => 'prepare for taking an online backup', proname => 'pg_backup_start', provolatile => 'v', proparallel => 'r', - prorettype => 'pg_lsn', proargtypes => 'text bool', + prorettype => 'record', proargtypes => 'text bool', + proallargtypes => '{text,bool,pg_lsn,int8,timestamptz}', + proargmodes => '{i,i,o,o,o}', + proargnames => '{label,fast,lsn,timeline_id,start}', prosrc => 'pg_backup_start' }, { oid => '2739', descr => 'finish taking an online backup', proname => 'pg_backup_stop', provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => 'bool', - proallargtypes => '{bool,pg_lsn,text,text}', proargmodes => '{i,o,o,o}', - proargnames => '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes => '{bool,bytea,text,pg_lsn,int8,timestamptz}', + proargmodes => '{i,o,o,o,o,o}', + proargnames => '{wait_for_archive,pg_control_file,tablespace_map_file,lsn,timeline_id,stop}', prosrc => 'pg_backup_stop' }, { oid => '3436', descr => 'promote standby server', proname => 'pg_promote', provolatile => 'v', prorettype => 'bool', @@ -11917,9 +11921,9 @@ { oid => '3443', descr => 'pg_controldata recovery state information as a function', proname => 'pg_control_recovery', provolatile => 'v', prorettype => 'record', - proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,bool}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes => '', proallargtypes => '{pg_lsn,int4,pg_lsn,pg_lsn,int4,pg_lsn,bool,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o}', + proargnames => '{min_recovery_end_lsn,min_recovery_end_timeline,backup_checkpoint_lsn,backup_start_lsn,backup_start_tli,backup_end_lsn,backup_recovery_required,backup_from_standby,end_of_backup_record_required}', prosrc => 'pg_control_recovery' }, { oid => '3444', -- 2.34.1 [application/pgp-signature] signature.asc (833B, ../../[email protected]/4-signature.asc) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-20 13:50 David Steele <[email protected]> parent: Michael Paquier <[email protected]> 2 siblings, 0 replies; 11+ messages in thread From: David Steele @ 2023-11-20 13:50 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: Robert Haas <[email protected]>; Postgres hackers <[email protected]> On 11/19/23 21:15, Michael Paquier wrote: > (I am not exactly sure how, but we've lost pgsql-hackers on the way > when you sent v5. Now added back in CC with the two latest patches > you've proposed attached.) Ugh, I must have hit reply instead of reply all. It's a rookie error and you hate to see it. > Here is a short summary of what has been missed by the lists: > - I've commented that the patch should not create, not show up in > fields returned the SQL functions or stream control files with a size > of 512B, just stick to 8kB. If this is worth changing this should be > applied consistently across the board including initdb, discussed on > its own thread. > - The backup-related fields in the control file are reset at the end > of recovery. I've suggested to not do that to keep a trace of what > was happening during recovery. The latest version of the patch resets > the fields. > - With the backup_label file gone, we lose some information in the > backups themselves, which is not good. Instead, you have suggested an > approach where this data is added to the backup manifest, meaning that > no information would be lost, particularly useful for self-contained > backups. The fields planned to be added to the backup manifest are: > -- The start and end time of the backup, the end timestamp being > useful to know when stop time can be used for PITR. > -- The backup label. > I've agreed that it may be the best thing to do at this end to not > lose any data related to the removal of the backup_label file. This looks right to me. > On Sun, Nov 19, 2023 at 02:14:32PM -0400, David Steele wrote: >> On 11/15/23 20:03, Michael Paquier wrote: >>> As the label is only an informational field, the parsing added to >>> pg_verifybackup is not really needed because it is used nowhere in the >>> validation process, so keeping the logic simpler would be the way to >>> go IMO. This is contrary to the WAL range for example, where start >>> and end LSNs are used for validation with a pg_waldump command. >>> Robert, any comments about the addition of the label in the manifest? >> >> I'm sure Robert will comment on this when he gets the time, but for now I >> have backed off on passing the new info to pg_verifybackup and added >> start/stop time. > > FWIW, I'm OK with the bits for the backup manifest as presented. So > if there are no remarks and/or no objections, I'd like to apply it but > let give some room to others to comment on that as there's been a gap > in the emails exchanged on pgsql-hackers. I hope that the summary > I've posted above covers everything. So let's see about doing > something around the middle of next week. With Thanksgiving in the > US, a lot of folks will not have the time to monitor what's happening > on this thread. Timing sounds good to me. > > + The end time for the backup. This is when the backup was stopped in > + <productname>PostgreSQL</productname> and represents the earliest time > + that can be used for time-based Point-In-Time Recovery. > > This one is actually a very good point. We'd lost this capacity with > the backup_label file gone without the end timestamps in the control > file. Yeah, the end time is very important for recovery scenarios. We definitely need that recorded somewhere. > I've noticed on the other thread the remark about being less > aggressive with the fields related to recovery in the control file, so > I assume that this patch should leave the fields be after the end of > recovery from the start and only rely on backupRecoveryRequired to > decide if the recovery should use the fields or not: > https://www.postgresql.org/message-id/241ccde1-1928-4ba2-a0bb-5350f7b191a8@=pgmasters.net > > + ControlFile->backupCheckPoint = InvalidXLogRecPtr; > ControlFile->backupStartPoint = InvalidXLogRecPtr; > + ControlFile->backupStartPointTLI = 0; > ControlFile->backupEndPoint = InvalidXLogRecPtr; > + ControlFile->backupFromStandby = false; > ControlFile->backupEndRequired = false; > > Still, I get the temptation of being consistent with the current style > on HEAD to reset everything, as well.. I'd rather reset everything for now (as we do now) and think about keeping these values as a separate patch. It may be that we don't want to keep all of them, or we need a separate flag to say recovery was completed. We are accumulating a lot of booleans here, maybe we need a state var (recoveryRequired, recoveryInProgress, recoveryComplete) and then define which other vars are valid in each state. Regards, -David ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-20 16:11 Robert Haas <[email protected]> parent: Michael Paquier <[email protected]> 2 siblings, 1 reply; 11+ messages in thread From: Robert Haas @ 2023-11-20 16:11 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: David Steele <[email protected]>; Postgres hackers <[email protected]> On Sun, Nov 19, 2023 at 8:16 PM Michael Paquier <[email protected]> wrote: > (I am not exactly sure how, but we've lost pgsql-hackers on the way > when you sent v5. Now added back in CC with the two latest patches > you've proposed attached.) > > Here is a short summary of what has been missed by the lists: > - I've commented that the patch should not create, not show up in > fields returned the SQL functions or stream control files with a size > of 512B, just stick to 8kB. If this is worth changing this should be > applied consistently across the board including initdb, discussed on > its own thread. > - The backup-related fields in the control file are reset at the end > of recovery. I've suggested to not do that to keep a trace of what > was happening during recovery. The latest version of the patch resets > the fields. > - With the backup_label file gone, we lose some information in the > backups themselves, which is not good. Instead, you have suggested an > approach where this data is added to the backup manifest, meaning that > no information would be lost, particularly useful for self-contained > backups. The fields planned to be added to the backup manifest are: > -- The start and end time of the backup, the end timestamp being > useful to know when stop time can be used for PITR. > -- The backup label. > I've agreed that it may be the best thing to do at this end to not > lose any data related to the removal of the backup_label file. I think we need more votes to make a change this big. I have a concern, which I think I've expressed before, that we keep whacking around the backup APIs, and that has a cost which is potentially larger than the benefits. The last time we changed the API, we changed pg_stop_backup to pg_backup_stop, but this doesn't do that, and I wonder if that's OK. Even if it is, do we really want to change this API around again after such a short time? That said, I don't have an intrinsic problem with moving this information from the backup_label to the backup_manifest file since it is purely informational. I do think there should perhaps be some additions to the test cases, though. I am concerned about the interaction of this proposal with incremental backup. When you take an incremental backup, you get something that looks a lot like a usable data directory but isn't. To prevent that from causing avoidable disasters, the present version of the patch adds INCREMENTAL FROM LSN and INCREMENTAL FROM TLI fields to the backup_label. pg_combinebackup knows to look for those fields, and the server knows that if they are present it should refuse to start. With this change, though, I suppose those fields would end up in pg_control. But that does not feel entirely great, because we have a goal of keeping the amount of real data in pg_control below 512 bytes, the traditional sector size, and this adds another 12 bytes of stuff to that file that currently doesn't need to be there. I feel like that's kind of a problem. But my main point here is ... if we have a few more senior hackers weigh in and vote in favor of this change, well then that's one thing. But IMHO a discussion that's mostly between 2 people is not nearly a strong enough consensus to justify this amount of disruption. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2023-11-20 23:37 Michael Paquier <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 11+ messages in thread From: Michael Paquier @ 2023-11-20 23:37 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: David Steele <[email protected]>; Postgres hackers <[email protected]> On Mon, Nov 20, 2023 at 11:11:13AM -0500, Robert Haas wrote: > I think we need more votes to make a change this big. I have a > concern, which I think I've expressed before, that we keep whacking > around the backup APIs, and that has a cost which is potentially > larger than the benefits. The last time we changed the API, we changed > pg_stop_backup to pg_backup_stop, but this doesn't do that, and I > wonder if that's OK. Even if it is, do we really want to change this > API around again after such a short time? Agreed. > That said, I don't have an intrinsic problem with moving this > information from the backup_label to the backup_manifest file since it > is purely informational. I do think there should perhaps be some > additions to the test cases, though. Yep, cool. Even if we decide to not go with what's discussed in this patch, I think that's useful for some users at the end to get more redundancy, as well. And that's in a format easier to parse. > I am concerned about the interaction of this proposal with incremental > backup. When you take an incremental backup, you get something that > looks a lot like a usable data directory but isn't. To prevent that > from causing avoidable disasters, the present version of the patch > adds INCREMENTAL FROM LSN and INCREMENTAL FROM TLI fields to the > backup_label. pg_combinebackup knows to look for those fields, and the > server knows that if they are present it should refuse to start. With > this change, though, I suppose those fields would end up in > pg_control. But that does not feel entirely great, because we have a > goal of keeping the amount of real data in pg_control below 512 bytes, > the traditional sector size, and this adds another 12 bytes of stuff > to that file that currently doesn't need to be there. I feel like > that's kind of a problem. I don't recall one time where the addition of new fields to the control file was easy to discuss because of its 512B hard limit. Anyway, putting the addition aside for a second, and I've not looked at the incremental backup patch, does the removal of the backup_label make the combine logic more complicated, or that's just moving a chunk of code to do a control file lookup instead of a backup_file parsing? Making the information less readable is definitely an issue for me. A different alternative that I've mentioned upthread is to keep an equivalent of the backup_label and rename it to something like backup.debug or similar, with a name good enough to tell people that we don't care about it being removed. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2024-01-26 12:57 vignesh C <[email protected]> parent: Michael Paquier <[email protected]> 2 siblings, 1 reply; 11+ messages in thread From: vignesh C @ 2024-01-26 12:57 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: David Steele <[email protected]>; Robert Haas <[email protected]>; Postgres hackers <[email protected]> On Mon, 20 Nov 2023 at 06:46, Michael Paquier <[email protected]> wrote: > > (I am not exactly sure how, but we've lost pgsql-hackers on the way > when you sent v5. Now added back in CC with the two latest patches > you've proposed attached.) > > Here is a short summary of what has been missed by the lists: > - I've commented that the patch should not create, not show up in > fields returned the SQL functions or stream control files with a size > of 512B, just stick to 8kB. If this is worth changing this should be > applied consistently across the board including initdb, discussed on > its own thread. > - The backup-related fields in the control file are reset at the end > of recovery. I've suggested to not do that to keep a trace of what > was happening during recovery. The latest version of the patch resets > the fields. > - With the backup_label file gone, we lose some information in the > backups themselves, which is not good. Instead, you have suggested an > approach where this data is added to the backup manifest, meaning that > no information would be lost, particularly useful for self-contained > backups. The fields planned to be added to the backup manifest are: > -- The start and end time of the backup, the end timestamp being > useful to know when stop time can be used for PITR. > -- The backup label. > I've agreed that it may be the best thing to do at this end to not > lose any data related to the removal of the backup_label file. > > On Sun, Nov 19, 2023 at 02:14:32PM -0400, David Steele wrote: > > On 11/15/23 20:03, Michael Paquier wrote: > >> As the label is only an informational field, the parsing added to > >> pg_verifybackup is not really needed because it is used nowhere in the > >> validation process, so keeping the logic simpler would be the way to > >> go IMO. This is contrary to the WAL range for example, where start > >> and end LSNs are used for validation with a pg_waldump command. > >> Robert, any comments about the addition of the label in the manifest? > > > > I'm sure Robert will comment on this when he gets the time, but for now I > > have backed off on passing the new info to pg_verifybackup and added > > start/stop time. > > FWIW, I'm OK with the bits for the backup manifest as presented. So > if there are no remarks and/or no objections, I'd like to apply it but > let give some room to others to comment on that as there's been a gap > in the emails exchanged on pgsql-hackers. I hope that the summary > I've posted above covers everything. So let's see about doing > something around the middle of next week. With Thanksgiving in the > US, a lot of folks will not have the time to monitor what's happening > on this thread. > > + The end time for the backup. This is when the backup was stopped in > + <productname>PostgreSQL</productname> and represents the earliest time > + that can be used for time-based Point-In-Time Recovery. > > This one is actually a very good point. We'd lost this capacity with > the backup_label file gone without the end timestamps in the control > file. > > > New patches attached based on b218fbb7. > > I've noticed on the other thread the remark about being less > aggressive with the fields related to recovery in the control file, so > I assume that this patch should leave the fields be after the end of > recovery from the start and only rely on backupRecoveryRequired to > decide if the recovery should use the fields or not: > https://www.postgresql.org/message-id/241ccde1-1928-4ba2-a0bb-5350f7b191a8@=pgmasters.net > > + ControlFile->backupCheckPoint = InvalidXLogRecPtr; > ControlFile->backupStartPoint = InvalidXLogRecPtr; > + ControlFile->backupStartPointTLI = 0; > ControlFile->backupEndPoint = InvalidXLogRecPtr; > + ControlFile->backupFromStandby = false; > ControlFile->backupEndRequired = false; > > Still, I get the temptation of being consistent with the current style > on HEAD to reset everything, as well.. CFBot shows that the patch does not apply anymore as in [1]: === Applying patches on top of PostgreSQL commit ID 7014c9a4bba2d1b67d60687afb5b2091c1d07f73 === === applying patch ./recovery-in-pgcontrol-v7-0001-add-info-to-manifest.patch patching file doc/src/sgml/backup-manifest.sgml patching file src/backend/backup/backup_manifest.c patching file src/backend/backup/basebackup.c Hunk #1 succeeded at 238 (offset 13 lines). Hunk #2 succeeded at 258 (offset 13 lines). Hunk #3 succeeded at 399 (offset 17 lines). Hunk #4 succeeded at 652 (offset 17 lines). can't find file to patch at input line 219 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |diff --git a/src/bin/pg_verifybackup/parse_manifest.c b/src/bin/pg_verifybackup/parse_manifest.c |index bf0227c668..408af88e58 100644 |--- a/src/bin/pg_verifybackup/parse_manifest.c |+++ b/src/bin/pg_verifybackup/parse_manifest.c -------------------------- No file to patch. Skipping patch. 9 out of 9 hunks ignored patching file src/include/backup/backup_manifest.h Please post an updated version for the same. [1] - http://cfbot.cputube.org/patch_46_3511.log Regards, Vignesh ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2024-01-28 23:11 Michael Paquier <[email protected]> parent: vignesh C <[email protected]> 0 siblings, 1 reply; 11+ messages in thread From: Michael Paquier @ 2024-01-28 23:11 UTC (permalink / raw) To: vignesh C <[email protected]>; +Cc: David Steele <[email protected]>; Robert Haas <[email protected]>; Postgres hackers <[email protected]> On Fri, Jan 26, 2024 at 06:27:30PM +0530, vignesh C wrote: > Please post an updated version for the same. > > [1] - http://cfbot.cputube.org/patch_46_3511.log With the recent introduction of incremental backups that depend on backup_label and the rather negative feedback received, I think that it would be better to return this entry as RwF for now. What do you think? -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Add recovery to pg_control and remove backup_label @ 2024-01-28 23:28 David Steele <[email protected]> parent: Michael Paquier <[email protected]> 0 siblings, 0 replies; 11+ messages in thread From: David Steele @ 2024-01-28 23:28 UTC (permalink / raw) To: Michael Paquier <[email protected]>; vignesh C <[email protected]>; +Cc: Robert Haas <[email protected]>; Postgres hackers <[email protected]>; Andres Freund <[email protected]>; Stephen Frost <[email protected]> On 1/28/24 19:11, Michael Paquier wrote: > On Fri, Jan 26, 2024 at 06:27:30PM +0530, vignesh C wrote: >> Please post an updated version for the same. >> >> [1] - http://cfbot.cputube.org/patch_46_3511.log > > With the recent introduction of incremental backups that depend on > backup_label and the rather negative feedback received, I think that > it would be better to return this entry as RwF for now. What do you > think? I've been thinking it makes little sense to update the patch. It would be a lot of work with all the new changes for incremental backup and since Andres and Robert appear to be very against the idea, I doubt it would be worth the effort. I have withdrawn the patch. Regards, -David ^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2024-01-28 23:28 UTC | newest] Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-05-28 18:16 [PATCH v30 4/6] Track collation versions for indexes. Thomas Munro <[email protected]> 2023-11-10 04:37 Re: Add recovery to pg_control and remove backup_label Michael Paquier <[email protected]> 2023-11-10 18:55 ` Re: Add recovery to pg_control and remove backup_label David Steele <[email protected]> 2023-11-13 04:36 ` Re: Add recovery to pg_control and remove backup_label Michael Paquier <[email protected]> 2023-11-20 01:15 ` Re: Add recovery to pg_control and remove backup_label Michael Paquier <[email protected]> 2023-11-20 13:50 ` Re: Add recovery to pg_control and remove backup_label David Steele <[email protected]> 2023-11-20 16:11 ` Re: Add recovery to pg_control and remove backup_label Robert Haas <[email protected]> 2023-11-20 23:37 ` Re: Add recovery to pg_control and remove backup_label Michael Paquier <[email protected]> 2024-01-26 12:57 ` Re: Add recovery to pg_control and remove backup_label vignesh C <[email protected]> 2024-01-28 23:11 ` Re: Add recovery to pg_control and remove backup_label Michael Paquier <[email protected]> 2024-01-28 23:28 ` Re: Add recovery to pg_control and remove backup_label David Steele <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox