Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r4std-00AjMU-26 for pgsql-hackers@arkaria.postgresql.org; Mon, 20 Nov 2023 01:16:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r4stZ-005M5e-Gz for pgsql-hackers@arkaria.postgresql.org; Mon, 20 Nov 2023 01:16:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r4stX-005M5W-JJ for pgsql-hackers@lists.postgresql.org; Mon, 20 Nov 2023 01:16:13 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r4stQ-007KgK-Uk for pgsql-hackers@lists.postgresql.org; Mon, 20 Nov 2023 01:16:10 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id E956B5C0118; Sun, 19 Nov 2023 20:16:02 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Sun, 19 Nov 2023 20:16:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm1; t=1700442962; x=1700529362; bh=qf /tA1Z1oE3Xr7SZzXJKjjLJaagzPI4I19hB3rWbXRM=; b=MNTl45o9T9a+lrzwd5 v2YLe/fPsdF5nIhao+R1+eFb6ApcpigHrjNnYtialkxnvbzZ6upVBELZN76OsxYC /mWb8xgK3nIaQ0nslBJsXRFw1yEEp3V75HuPshtGutx5Sh68usa18WthJFPZHav4 vfC/oqXd4GSq4WxDpJCt26Hl1AH5WbHHmiZugWNARNbb1v9BBmqsi3CGyVNT5QVZ WUquT3PhEb/80yTfIdN1zbkAQBSDE8VpOy4JgzDeWGszAVDLumY2V48+Qs5i9rsf nM0jQVw3cBXLGs7X4pFShAJdGNVcQqRmDo/foEuTCt0hO/HL10Q8dvSz9zkYoUCX ubpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1700442962; x=1700529362; bh=qf/tA1Z1oE3Xr 7SZzXJKjjLJaagzPI4I19hB3rWbXRM=; b=DqCff3XCWFJOaffXC+hQXzwQmqIKU aj5V9w3HjQEcTn/V8f1l+joug80cq3vYcyfi5Uy5yerg/SuTqZMkjX9nOalepqAC yeOx4WBZj23GwCanhUT0dso9i5kzBkI2LEY5yGIxKzP7Qv5HeXUbEVfYFKqQmWGB 7mWzD/sV0QBPfvzoJNJtsC5MSQRGmrw5Mqo/PkjWCfCDTa+pYyozPBoNMMjS7fut YIh7MhCkV9+Yb89xy9Py4JdrY6hK175BiGJN+KozgYOYSZTCvgZCZItviqxGqfbU u2R56lhh8nF6fZDLDEQxEr20aBqhj4U1OUbqF36bTxfeW1sajbXChE2iQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudeghedgfeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne gfrhhlucfvnfffucdlfeehmdenucfjughrpeffhffvvefukfhfgggtuggjsehgtderredt tddvnecuhfhrohhmpefoihgthhgrvghlucfrrghquhhivghruceomhhitghhrggvlhesph grqhhuihgvrhdrgiihiieqnecuggftrfgrthhtvghrnhepudduffekhfevgeehfeeifeeh keekudeljeevgedutdetieetledttdehleegfeeinecuffhomhgrihhnpehpohhsthhgrh gvshhqlhdrohhrghdpphhgmhgrshhtvghrshdrnhgvthenucevlhhushhtvghrufhiiigv pedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmihgthhgrvghlsehprghquhhivghrrd ighiii X-ME-Proxy: Feedback-ID: i0fe9450f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 19 Nov 2023 20:16:00 -0500 (EST) Date: Mon, 20 Nov 2023 10:15:49 +0900 From: Michael Paquier To: David Steele Cc: Robert Haas , Postgres hackers Subject: Re: Add recovery to pg_control and remove backup_label Message-ID: References: <46c300b9-96a1-449c-8ecd-9003bd02cb6b@pgmasters.net> <77ad92e7-3e37-40e2-8e3b-df3cf0108aa8@pgmasters.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="jTQITlajykV8t6gp" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --jTQITlajykV8t6gp Content-Type: multipart/mixed; boundary="W0+6J45DoLYiQ+WQ" Content-Disposition: inline --W0+6J45DoLYiQ+WQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable (I am not exactly sure how, but we've lost pgsql-hackers on the way when you sent v5. Now added back in CC with the two latest patches you've proposed attached.) Here is a short summary of what has been missed by the lists: - I've commented that the patch should not create, not show up in fields returned the SQL functions or stream control files with a size of 512B, just stick to 8kB. If this is worth changing this should be applied consistently across the board including initdb, discussed on its own thread. - The backup-related fields in the control file are reset at the end of recovery. I've suggested to not do that to keep a trace of what was happening during recovery. The latest version of the patch resets the fields. - With the backup_label file gone, we lose some information in the backups themselves, which is not good. Instead, you have suggested an approach where this data is added to the backup manifest, meaning that no information would be lost, particularly useful for self-contained backups. The fields planned to be added to the backup manifest are: -- The start and end time of the backup, the end timestamp being useful to know when stop time can be used for PITR. -- The backup label. I've agreed that it may be the best thing to do at this end to not lose any data related to the removal of the backup_label file. On Sun, Nov 19, 2023 at 02:14:32PM -0400, David Steele wrote: > On 11/15/23 20:03, Michael Paquier wrote: >> As the label is only an informational field, the parsing added to >> pg_verifybackup is not really needed because it is used nowhere in the >> validation process, so keeping the logic simpler would be the way to >> go IMO. This is contrary to the WAL range for example, where start >> and end LSNs are used for validation with a pg_waldump command. >> Robert, any comments about the addition of the label in the manifest? > > I'm sure Robert will comment on this when he gets the time, but for now I > have backed off on passing the new info to pg_verifybackup and added > start/stop time. FWIW, I'm OK with the bits for the backup manifest as presented. So if there are no remarks and/or no objections, I'd like to apply it but let give some room to others to comment on that as there's been a gap in the emails exchanged on pgsql-hackers. I hope that the summary I've posted above covers everything. So let's see about doing something around the middle of next week. With Thanksgiving in the US, a lot of folks will not have the time to monitor what's happening on this thread. + The end time for the backup. This is when the backup was stopped in + PostgreSQL and represents the earliest ti= me + that can be used for time-based Point-In-Time Recovery. This one is actually a very good point. We'd lost this capacity with the backup_label file gone without the end timestamps in the control file. > New patches attached based on b218fbb7. I've noticed on the other thread the remark about being less aggressive with the fields related to recovery in the control file, so I assume that this patch should leave the fields be after the end of recovery from the start and only rely on backupRecoveryRequired to decide if the recovery should use the fields or not: https://www.postgresql.org/message-id/241ccde1-1928-4ba2-a0bb-5350f7b191a8@= =3Dpgmasters.net + ControlFile->backupCheckPoint =3D InvalidXLogRecPtr; ControlFile->backupStartPoint =3D InvalidXLogRecPtr; + ControlFile->backupStartPointTLI =3D 0; ControlFile->backupEndPoint =3D InvalidXLogRecPtr; + ControlFile->backupFromStandby =3D false; ControlFile->backupEndRequired =3D false; Still, I get the temptation of being consistent with the current style on HEAD to reset everything, as well..=20 -- Michael --W0+6J45DoLYiQ+WQ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="recovery-in-pgcontrol-v7-0001-add-info-to-manifest.patch" Content-Transfer-Encoding: quoted-printable =46rom 97bb113b5bf5427449b748c3ee25b647a2c5fef5 Mon Sep 17 00:00:00 2001 =46rom: David Steele Date: Sun, 19 Nov 2023 16:54:36 +0000 Subject: Add label and start/stop time to backup manifest. Add label passed by the user to pg_basebackup and backup start/stop time to the backup_manifest file. Currently these fields are purely for information= al purposes. --- doc/src/sgml/backup-manifest.sgml | 51 +++++++++++++++++++++ src/backend/backup/backup_manifest.c | 56 +++++++++++++++++++++++- src/backend/backup/basebackup.c | 9 +++- src/bin/pg_verifybackup/parse_manifest.c | 40 +++++++++++++++++ src/include/backup/backup_manifest.h | 5 ++- 5 files changed, 157 insertions(+), 4 deletions(-) diff --git a/doc/src/sgml/backup-manifest.sgml b/doc/src/sgml/backup-manife= st.sgml index 771be1310a..a80b79e587 100644 --- a/doc/src/sgml/backup-manifest.sgml +++ b/doc/src/sgml/backup-manifest.sgml @@ -42,6 +42,16 @@ =20 + + Backup-Label + + + Backup label specified by the user. This will be set to a default va= lue + if no backup label was specified. + + + + Files @@ -67,6 +77,17 @@ =20 + + Time-Range + + + The associated value is always an object describing the start/stop t= ime + of the backup. The structure of this object is further described in + . + + + + Manifest-Checksum @@ -213,4 +234,34 @@ for the same timeline. + + + Backup Manifest Time Range Object + + + The object which describes the time range always has three keys: + + + + + Start-Time + + + The start time for the backup. + + + + + + End-Time + + + The end time for the backup. This is when the backup was stopped in + PostgreSQL and represents the earliest ti= me + that can be used for time-based Point-In-Time Recovery. + + + + + diff --git a/src/backend/backup/backup_manifest.c b/src/backend/backup/back= up_manifest.c index aeed362a9a..9540f05964 100644 --- a/src/backend/backup/backup_manifest.c +++ b/src/backend/backup/backup_manifest.c @@ -56,7 +56,8 @@ IsManifestEnabled(backup_manifest_info *manifest) void InitializeBackupManifest(backup_manifest_info *manifest, backup_manifest_option want_manifest, - pg_checksum_type manifest_checksum_type) + pg_checksum_type manifest_checksum_type, + const char *label) { memset(manifest, 0, sizeof(backup_manifest_info)); manifest->checksum_type =3D manifest_checksum_type; @@ -78,9 +79,21 @@ InitializeBackupManifest(backup_manifest_info *manifest, manifest->still_checksumming =3D true; =20 if (want_manifest !=3D MANIFEST_OPTION_NO) + { + StringInfoData buf; + AppendToManifest(manifest, "{ \"PostgreSQL-Backup-Manifest-Version\": 1,\n" - "\"Files\": ["); + "\"Backup-Label\": "); + + /* JSON encode label and add it to manifest */ + initStringInfo(&buf); + escape_json(&buf, label); + AppendStringToManifest(manifest, buf.data); + pfree(buf.data); + + AppendToManifest(manifest, ",\n\"Files\": ["); + } } =20 /* @@ -308,6 +321,45 @@ AddWALInfoToBackupManifest(backup_manifest_info *manif= est, XLogRecPtr startptr, AppendStringToManifest(manifest, "\n],\n"); } =20 +/* + * Add backup start/end time information to the manifest. + */ +void +AddTimeInfoToBackupManifest(backup_manifest_info *manifest, pg_time_t star= ttime, + pg_time_t endtime) +{ + StringInfoData buf; + + if (!IsManifestEnabled(manifest)) + return; + + /* Start the time range. */ + AppendStringToManifest(manifest, "\"Time-Range\": { "); + + /* + * Convert start/end time to strings and append them to the manifest. Sin= ce + * it's not clear what time zone to use and since time zone definitions c= an + * change, possibly causing confusion, use GMT always. + */ + initStringInfo(&buf); + + appendStringInfoString(&buf, "\"Start-Time\": \""); + enlargeStringInfo(&buf, 128); + buf.len +=3D pg_strftime(&buf.data[buf.len], 128, "%Y-%m-%d %H:%M:%S %Z", + pg_gmtime(&starttime)); + appendStringInfoString(&buf, "\", \"End-Time\": \""); + enlargeStringInfo(&buf, 128); + buf.len +=3D pg_strftime(&buf.data[buf.len], 128, "%Y-%m-%d %H:%M:%S %Z", + pg_gmtime(&endtime)); + appendStringInfoString(&buf, "\" },\n"); + + /* Add to the manifest. */ + AppendStringToManifest(manifest, buf.data); + + /* Avoid leaking memory. */ + pfree(buf.data); +} + /* * Finalize the backup manifest, and send it to the client. */ diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebacku= p.c index 35dd79babc..c7b3ba3e6e 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -225,6 +225,8 @@ perform_base_backup(basebackup_options *opt, bbsink *si= nk) bbsink_state state; XLogRecPtr endptr; TimeLineID endtli; + pg_time_t starttime; + pg_time_t stoptime; backup_manifest_info manifest; BackupState *backup_state; StringInfo tablespace_map; @@ -243,7 +245,7 @@ perform_base_backup(basebackup_options *opt, bbsink *si= nk) backup_started_in_recovery =3D RecoveryInProgress(); =20 InitializeBackupManifest(&manifest, opt->manifest, - opt->manifest_checksum_type); + opt->manifest_checksum_type, opt->label); =20 total_checksum_failures =3D 0; =20 @@ -380,6 +382,10 @@ perform_base_backup(basebackup_options *opt, bbsink *s= ink) endptr =3D backup_state->stoppoint; endtli =3D backup_state->stoptli; =20 + /* Record start/stop time for manifest */ + starttime =3D backup_state->starttime; + stoptime =3D backup_state->stoptime; + /* Deallocate backup-related variables. */ pfree(tablespace_map->data); pfree(tablespace_map); @@ -629,6 +635,7 @@ perform_base_backup(basebackup_options *opt, bbsink *si= nk) =20 AddWALInfoToBackupManifest(&manifest, state.startptr, state.starttli, endptr, endtli); + AddTimeInfoToBackupManifest(&manifest, starttime, stoptime); =20 SendBackupManifest(&manifest, sink); =20 diff --git a/src/bin/pg_verifybackup/parse_manifest.c b/src/bin/pg_verifyba= ckup/parse_manifest.c index bf0227c668..408af88e58 100644 --- a/src/bin/pg_verifybackup/parse_manifest.c +++ b/src/bin/pg_verifybackup/parse_manifest.c @@ -25,6 +25,7 @@ typedef enum JM_EXPECT_TOPLEVEL_END, JM_EXPECT_TOPLEVEL_FIELD, JM_EXPECT_VERSION_VALUE, + JM_EXPECT_BACKUP_LABEL_VALUE, JM_EXPECT_FILES_START, JM_EXPECT_FILES_NEXT, JM_EXPECT_THIS_FILE_FIELD, @@ -33,6 +34,9 @@ typedef enum JM_EXPECT_WAL_RANGES_NEXT, JM_EXPECT_THIS_WAL_RANGE_FIELD, JM_EXPECT_THIS_WAL_RANGE_VALUE, + JM_EXPECT_TIME_RANGE_START, + JM_EXPECT_THIS_TIME_RANGE_FIELD, + JM_EXPECT_THIS_TIME_RANGE_VALUE, JM_EXPECT_MANIFEST_CHECKSUM_VALUE, JM_EXPECT_EOF, } JsonManifestSemanticState; @@ -188,6 +192,9 @@ json_manifest_object_start(void *state) parse->start_lsn =3D NULL; parse->end_lsn =3D NULL; break; + case JM_EXPECT_TIME_RANGE_START: + parse->state =3D JM_EXPECT_THIS_TIME_RANGE_FIELD; + break; default: json_manifest_parse_failure(parse->context, "unexpected object start"); @@ -223,6 +230,9 @@ json_manifest_object_end(void *state) json_manifest_finalize_wal_range(parse); parse->state =3D JM_EXPECT_WAL_RANGES_NEXT; break; + case JM_EXPECT_THIS_TIME_RANGE_FIELD: + parse->state =3D JM_EXPECT_TOPLEVEL_FIELD; + break; default: json_manifest_parse_failure(parse->context, "unexpected object end"); @@ -312,6 +322,13 @@ json_manifest_object_field_start(void *state, char *fn= ame, bool isnull) break; } =20 + /* Is this the backup label? */ + if (strcmp(fname, "Backup-Label") =3D=3D 0) + { + parse->state =3D JM_EXPECT_BACKUP_LABEL_VALUE; + break; + } + /* Is this the list of files? */ if (strcmp(fname, "Files") =3D=3D 0) { @@ -326,6 +343,13 @@ json_manifest_object_field_start(void *state, char *fn= ame, bool isnull) break; } =20 + /* Is this the time range? */ + if (strcmp(fname, "Time-Range") =3D=3D 0) + { + parse->state =3D JM_EXPECT_TIME_RANGE_START; + break; + } + /* Is this the manifest checksum? */ if (strcmp(fname, "Manifest-Checksum") =3D=3D 0) { @@ -372,6 +396,14 @@ json_manifest_object_field_start(void *state, char *fn= ame, bool isnull) parse->state =3D JM_EXPECT_THIS_WAL_RANGE_VALUE; break; =20 + case JM_EXPECT_THIS_TIME_RANGE_FIELD: + if (strcmp(fname, "Start-Time") !=3D 0 && + strcmp(fname, "End-Time") !=3D 0) + json_manifest_parse_failure(parse->context, + "unexpected time range field"); + parse->state =3D JM_EXPECT_THIS_TIME_RANGE_VALUE; + break; + default: json_manifest_parse_failure(parse->context, "unexpected object field"); @@ -410,6 +442,10 @@ json_manifest_scalar(void *state, char *token, JsonTok= enType tokentype) parse->state =3D JM_EXPECT_TOPLEVEL_FIELD; break; =20 + case JM_EXPECT_BACKUP_LABEL_VALUE: + parse->state =3D JM_EXPECT_TOPLEVEL_FIELD; + break; + case JM_EXPECT_THIS_FILE_VALUE: switch (parse->file_field) { @@ -451,6 +487,10 @@ json_manifest_scalar(void *state, char *token, JsonTok= enType tokentype) parse->state =3D JM_EXPECT_THIS_WAL_RANGE_FIELD; break; =20 + case JM_EXPECT_THIS_TIME_RANGE_VALUE: + parse->state =3D JM_EXPECT_THIS_TIME_RANGE_FIELD; + break; + case JM_EXPECT_MANIFEST_CHECKSUM_VALUE: parse->state =3D JM_EXPECT_TOPLEVEL_END; parse->manifest_checksum =3D token; diff --git a/src/include/backup/backup_manifest.h b/src/include/backup/back= up_manifest.h index bd7067ae42..4ab1291bba 100644 --- a/src/include/backup/backup_manifest.h +++ b/src/include/backup/backup_manifest.h @@ -37,7 +37,8 @@ typedef struct backup_manifest_info =20 extern void InitializeBackupManifest(backup_manifest_info *manifest, backup_manifest_option want_manifest, - pg_checksum_type manifest_checksum_type); + pg_checksum_type manifest_checksum_type, + const char *label); extern void AddFileToBackupManifest(backup_manifest_info *manifest, Oid spcoid, const char *pathname, size_t size, @@ -47,6 +48,8 @@ extern void AddWALInfoToBackupManifest(backup_manifest_in= fo *manifest, XLogRecPtr startptr, TimeLineID starttli, XLogRecPtr endptr, TimeLineID endtli); +extern void AddTimeInfoToBackupManifest(backup_manifest_info *manifest, + pg_time_t starttime, pg_time_t endtime); =20 extern void SendBackupManifest(backup_manifest_info *manifest, bbsink *sin= k); extern void FreeBackupManifest(backup_manifest_info *manifest); --=20 2.34.1 --W0+6J45DoLYiQ+WQ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="recovery-in-pgcontrol-v7-0002-remove-backuplabel.patch" Content-Transfer-Encoding: quoted-printable =46rom 6b732c6e085a4ef2ec717af943b44d02cb4b9849 Mon Sep 17 00:00:00 2001 =46rom: David Steele Date: Sun, 19 Nov 2023 16:54:37 +0000 Subject: Add recovery to pg_control and remove backup_label. Simplify and harden recovery by removing backup_label and storing recovery information directly in pg_control. Instead of backup software copying pg_control from PGDATA, it stores an updated version that is returned from pg_backup_stop(). This is better for the following reasons: * The user can no longer remove backup_label and get what looks like a successful recovery (while almost certainly causing corruption). If pg_cont= rol is removed the cluster will not start. The user may try pg_resetwal, but th= at tool makes it pretty clear that corruption will result from its use. We cou= ld also modify pg_resetwal to complain if recovery info is present in pg_contr= ol. * We don't need to worry about backup software seeing a torn copy of pg_con= trol, since Postgres can safely read it out of memory and provide a valid copy via pg_backup_stop(). This solves torn reads without needing to write pg_contro= l via a temp file, which may affect performance on a standby. * For backup from standby, we no longer need to instruct the backup softwar= e to copy pg_control last. In fact the backup software should not copy pg_contro= l from PGDATA at all. Since backup_label is now gone, the fields that used to be in backup_label = are now provided as columns returned from pg_backup_start() and pg_backup_stop(= ) and the backup history file is still written to the archive. For pg_basebackup,= the label passed on the cmd line as --label is now stored in the manifest. Control and catalog version bumps are required. --- doc/src/sgml/backup.sgml | 31 ++- doc/src/sgml/func.sgml | 41 ++- doc/src/sgml/ref/pg_rewind.sgml | 6 +- src/backend/access/transam/xlog.c | 67 ++--- src/backend/access/transam/xlogbackup.c | 39 ++- src/backend/access/transam/xlogfuncs.c | 50 ++-- src/backend/access/transam/xlogrecovery.c | 249 ++++--------------- src/backend/backup/basebackup.c | 37 +-- src/backend/catalog/system_functions.sql | 10 +- src/backend/utils/misc/pg_controldata.c | 22 +- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 15 +- src/bin/pg_controldata/pg_controldata.c | 8 + src/bin/pg_resetwal/pg_resetwal.c | 4 + src/bin/pg_rewind/filemap.c | 7 +- src/bin/pg_rewind/pg_rewind.c | 64 +---- src/include/access/xlog.h | 2 - src/include/access/xlogbackup.h | 12 +- src/include/access/xlogrecovery.h | 3 +- src/include/catalog/pg_control.h | 19 +- src/include/catalog/pg_proc.dat | 16 +- 20 files changed, 293 insertions(+), 409 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 8cb24d6ae5..584384875b 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -935,19 +935,20 @@ SELECT * FROM pg_backup_stop(wait_for_archive =3D> tr= ue); ready to archive. - pg_backup_stop will return one row with three - values. The second of these fields should be written to a file named - backup_label in the root directory of the backup= =2E The - third field should be written to a file named - tablespace_map unless the field is empty. These = files are + pg_backup_stop returns the + pg_control file, which must be stored in the + global directory of the backup. It also returns = the + tablespace_map file, which should be written in = the + root directory of the backup unless the field is empty. These files a= re vital to the backup working and must be written byte for byte without - modification, which may require opening the file in binary mode. + modification, which will require opening the file in binary mode. Once the WAL segment files active during the backup are archived, you= are - done. The file identified by pg_backup_stop's f= irst return + done. The file identified by pg_backup_stop's + lsn return value is the last segment that is required to form a complete set of backup files. On a primary, if archive_mode is en= abled and the wait_for_archive parameter is true, @@ -1013,7 +1014,15 @@ SELECT * FROM pg_backup_stop(wait_for_archive =3D> t= rue); =20 - You should, however, omit from the backup the files within the + You must exclude global/pg_control from your back= up + and put the contents of the pg_control_file col= umn + returned from pg_backup_stop in your backup at + global/pg_control. This file contains the informa= tion + required to safely recover. + + + + You should also omit from the backup the files within the cluster's pg_wal/ subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if @@ -1062,11 +1071,11 @@ SELECT * FROM pg_backup_stop(wait_for_archive =3D> = true); =20 - The backup label - file includes the label string you gave to pg_backup_start, + The backup history file (which is archived like WAL) includes the label + string you gave to pg_backup_start, as well as the time at which pg_backup_start was = run, and the name of the starting WAL file. In case of confusion it is therefo= re - possible to look inside a backup file and determine exactly which + possible to look inside a backup history file and determine exactly wh= ich backup session the dump file came from. The tablespace map file inclu= des the symbolic link names as they exist in the directory pg_tblspc/ and the full path of each symbolic lin= k. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 93f068edcf..5f27cce161 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26865,7 +26865,10 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 62= 2360 free (88 chunks); 1029560 label text , fast boolean ) - pg_lsn + record + ( lsn pg_lsn, + timeline_id int8, + start timestamptz ) Prepares the server to begin an on-line backup. The only required @@ -26877,6 +26880,13 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 62= 2360 free (88 chunks); 1029560 as possible. This forces an immediate checkpoint which will cause= a spike in I/O operations, slowing any concurrently executing querie= s. + + The result columns contain information about the start of the back= up + and can be ignored: the lsn column holds the + starting write-ahead log location, the + timeline_id column holds the starting timel= ine, + and the stop column holds the starting time= stamp. + This function is restricted to superusers by default, but other us= ers can be granted EXECUTE to run the function. @@ -26892,13 +26902,15 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 6= 22360 free (88 chunks); 1029560 wait_for_archive boolean<= /type> ) record - ( lsn pg_lsn, - labelfile text, - spcmapfile text ) + ( pg_control_file text, + tablespace_map_file text, + lsn pg_lsn, + timeline_id int8, + stop timestamptz ) Finishes performing an on-line backup. The desired contents of the - backup label file and the tablespace map file are returned as part= of + pg_control file and the tablespace map file are returned as part of the result of the function and must be written to files in the backup area. These files must not be written to the live data dir= ectory (doing so will cause PostgreSQL to fail to restart in the event of= a @@ -26930,13 +26942,18 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 6= 22360 free (88 chunks); 1029560 backup. - The result of the function is a single record. - The lsn column holds the backup's ending - write-ahead log location (which again can be ignored). The second - column returns the contents of the backup label file, and the third - column returns the contents of the tablespace map file. These mus= t be - stored as part of the backup and are required as part of the resto= re - process. + The result of the function is a single record. The first column re= turns + the contents of the pg_control file and the + second column returns the contents of the + tablespace_map file. These must be stored as= part + of the backup and are required as part of the restore process. Not= e that + pg_control will be 8192KiB to match the on-di= sk + size of pg_control. The remainder of the colu= mns + contain information about the end of the backup and can be ignored= : the + lsn column holds the ending write-ahead log + location, the timeline_id column holds the = ending + timeline, and the stop column holds the end= ing + timestamp. This function is restricted to superusers by default, but other us= ers diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.s= gml index 8e0000d39f..404ce7c65e 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -400,7 +400,6 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_fil= e(text, bigint, bigint, b pg_serial/, pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ are omitted from the data copied from the source cluster. The files - backup_label, tablespace_map, pg_internal.init, postmaster.opts, and @@ -410,9 +409,8 @@ GRANT EXECUTE ON function pg_catalog.pg_read_binary_fil= e(text, bigint, bigint, b - Create a backup_label file to begin WAL replay = at - the checkpoint created at failover and configure the - pg_control file with a minimum consistency LSN + Update pg_control file to begin WAL replay at + the checkpoint created at failover and with a minimum consistency LSN defined as the result of pg_current_wal_insert_lsn() when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source. diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam= /xlog.c index 1159dff1a6..928cad0651 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -74,6 +74,7 @@ #include "pg_trace.h" #include "pgstat.h" #include "port/atomics.h" +#include "port/pg_crc32c.h" #include "port/pg_iovec.h" #include "postmaster/bgwriter.h" #include "postmaster/startup.h" @@ -5135,7 +5136,6 @@ StartupXLOG(void) bool wasShutdown; bool didCrash; bool haveTblspcMap; - bool haveBackupLabel; XLogRecPtr EndOfLog; TimeLineID EndOfLogTLI; TimeLineID newTLI; @@ -5259,13 +5259,14 @@ StartupXLOG(void) /* * Prepare for WAL recovery if needed. * - * InitWalRecovery analyzes the control file and the backup label file, if - * any. It updates the in-memory ControlFile buffer according to the - * starting checkpoint, and sets InRecovery and ArchiveRecoveryRequested. + * InitWalRecovery analyzes the control file and checks if backup recovery + * has been requested. It updates the in-memory ControlFile buffer + * according to the starting checkpoint, and sets InRecovery and + * ArchiveRecoveryRequested. + * * It also applies the tablespace map file, if any. */ - InitWalRecovery(ControlFile, &wasShutdown, - &haveBackupLabel, &haveTblspcMap); + InitWalRecovery(ControlFile, &wasShutdown, &haveTblspcMap); checkPoint =3D ControlFile->checkPointCopy; =20 /* initialize shared memory variables from the checkpoint record */ @@ -5408,20 +5409,6 @@ StartupXLOG(void) */ UpdateControlFile(); =20 - /* - * If there was a backup label file, it's done its job and the info - * has now been propagated into pg_control. We must get rid of the - * label file so that if we crash during recovery, we'll pick up at - * the latest recovery restartpoint instead of going all the way back - * to the backup start point. It seems prudent though to just rename - * the file out of the way rather than delete it completely. - */ - if (haveBackupLabel) - { - unlink(BACKUP_LABEL_OLD); - durable_rename(BACKUP_LABEL_FILE, BACKUP_LABEL_OLD, FATAL); - } - /* * If there was a tablespace_map file, it's done its job and the * symlinks have been created. We must get rid of the map file so @@ -5571,10 +5558,8 @@ StartupXLOG(void) * (at which point we reset backupStartPoint to be Invalid), for * backup-from-replica (which can't inject records into the WAL stream), * that point is when we reach the minRecoveryPoint in pg_control (which - * we purposefully copy last when backing up from a replica). For - * pg_rewind (which creates a backup_label with a method of "pg_rewind") - * or snapshot-style backups (which don't), backupEndRequired will be set - * to false. + * we purposefully copy last when backing up). For pg_rewind or + * snapshot-style backups, backupEndRequired will be set to false. * * Note: it is indeed okay to look at the local variable * LocalMinRecoveryPoint here, even though ControlFile->minRecoveryPoint @@ -5951,8 +5936,11 @@ ReachedEndOfBackup(XLogRecPtr EndRecPtr, TimeLineID = tli) ControlFile->minRecoveryPointTLI =3D tli; } =20 + ControlFile->backupCheckPoint =3D InvalidXLogRecPtr; ControlFile->backupStartPoint =3D InvalidXLogRecPtr; + ControlFile->backupStartPointTLI =3D 0; ControlFile->backupEndPoint =3D InvalidXLogRecPtr; + ControlFile->backupFromStandby =3D false; ControlFile->backupEndRequired =3D false; UpdateControlFile(); =20 @@ -8744,11 +8732,33 @@ do_pg_backup_stop(BackupState *state, bool waitfora= rchive) int seconds_before_warning; int waits =3D 0; bool reported_waiting =3D false; + ControlFileData *controlFileCopy =3D (ControlFileData *)state->controlFil= e; =20 Assert(state !=3D NULL); =20 backup_stopped_in_recovery =3D RecoveryInProgress(); =20 + /* + * Create a copy of control data and update it with fields required for + * recovery. Also recalculate the CRC. + */ + memset(controlFileCopy, 0, PG_CONTROL_FILE_SIZE); + + LWLockAcquire(ControlFileLock, LW_SHARED); + memcpy(controlFileCopy, ControlFile, sizeof(ControlFileData)); + LWLockRelease(ControlFileLock); + + controlFileCopy->backupRecoveryRequired =3D true; + controlFileCopy->backupFromStandby =3D backup_stopped_in_recovery; + controlFileCopy->backupEndRequired =3D true; + controlFileCopy->backupCheckPoint =3D state->checkpointloc; + controlFileCopy->backupStartPoint =3D state->startpoint; + controlFileCopy->backupStartPointTLI =3D state->starttli; + + INIT_CRC32C(controlFileCopy->crc); + COMP_CRC32C(controlFileCopy->crc, controlFileCopy, offsetof(ControlFileDa= ta, crc)); + FIN_CRC32C(controlFileCopy->crc); + /* * During recovery, we don't need to check WAL level. Because, if WAL * level is not sufficient, it's impossible to get here during recovery. @@ -8850,11 +8860,8 @@ do_pg_backup_stop(BackupState *state, bool waitforar= chive) "Enable full_page_writes and run CHECKPOINT on the primary, " "and then try an online backup again."))); =20 - - LWLockAcquire(ControlFileLock, LW_SHARED); - state->stoppoint =3D ControlFile->minRecoveryPoint; - state->stoptli =3D ControlFile->minRecoveryPointTLI; - LWLockRelease(ControlFileLock); + state->stoppoint =3D controlFileCopy->minRecoveryPoint; + state->stoptli =3D controlFileCopy->minRecoveryPointTLI; } else { @@ -8896,7 +8903,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarc= hive) histfilepath))); =20 /* Build and save the contents of the backup history file */ - history_file =3D build_backup_content(state, true); + history_file =3D build_backup_history_content(state); fprintf(fp, "%s", history_file); pfree(history_file); =20 diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/t= ransam/xlogbackup.c index 21d68133ae..22c95f3c4c 100644 --- a/src/backend/access/transam/xlogbackup.c +++ b/src/backend/access/transam/xlogbackup.c @@ -18,19 +18,19 @@ #include "access/xlogbackup.h" =20 /* - * Build contents for backup_label or backup history file. - * - * When ishistoryfile is true, it creates the contents for a backup history - * file, otherwise it creates contents for a backup_label file. + * Build contents for backup history file. * * Returns the result generated as a palloc'd string. */ char * -build_backup_content(BackupState *state, bool ishistoryfile) +build_backup_history_content(BackupState *state) { char startstrbuf[128]; + char stopstrfbuf[128]; char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */ + char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ XLogSegNo startsegno; + XLogSegNo stopsegno; StringInfo result =3D makeStringInfo(); char *data; =20 @@ -45,16 +45,10 @@ build_backup_content(BackupState *state, bool ishistory= file) appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n", LSN_FORMAT_ARGS(state->startpoint), startxlogfile); =20 - if (ishistoryfile) - { - char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */ - XLogSegNo stopsegno; - - XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); - XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); - appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", - LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); - } + XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size); + XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size); + appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", + LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile); =20 appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n", LSN_FORMAT_ARGS(state->checkpointloc)); @@ -65,17 +59,12 @@ build_backup_content(BackupState *state, bool ishistory= file) appendStringInfo(result, "LABEL: %s\n", state->name); appendStringInfo(result, "START TIMELINE: %u\n", state->starttli); =20 - if (ishistoryfile) - { - char stopstrfbuf[128]; - - /* Use the log timezone here, not the session timezone */ - pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", - pg_localtime(&state->stoptime, log_timezone)); + /* Use the log timezone here, not the session timezone */ + pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z", + pg_localtime(&state->stoptime, log_timezone)); =20 - appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); - appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); - } + appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf); + appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli); =20 data =3D result->data; pfree(result); diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/tr= ansam/xlogfuncs.c index 45a70668b1..d81d36705f 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -53,7 +53,7 @@ static MemoryContext backupcontext =3D NULL; * pg_backup_start: set up for taking an on-line backup dump * * Essentially what this does is to create the contents required for the - * backup_label file and the tablespace map. + * the tablespace map. * * Permission checking for this function is managed through the normal * GRANT system. @@ -61,6 +61,10 @@ static MemoryContext backupcontext =3D NULL; Datum pg_backup_start(PG_FUNCTION_ARGS) { +#define PG_BACKUP_START_V2_COLS 3 + TupleDesc tupdesc; + Datum values[PG_BACKUP_START_V2_COLS] =3D {0}; + bool nulls[PG_BACKUP_START_V2_COLS] =3D {0}; text *backupid =3D PG_GETARG_TEXT_PP(0); bool fast =3D PG_GETARG_BOOL(1); char *backupidstr; @@ -69,6 +73,10 @@ pg_backup_start(PG_FUNCTION_ARGS) =20 backupidstr =3D text_to_cstring(backupid); =20 + /* Initialize attributes information in the tuple descriptor */ + if (get_call_result_type(fcinfo, NULL, &tupdesc) !=3D TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + if (status =3D=3D SESSION_BACKUP_RUNNING) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -102,7 +110,12 @@ pg_backup_start(PG_FUNCTION_ARGS) register_persistent_abort_backup_handler(); do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map); =20 - PG_RETURN_LSN(backup_state->startpoint); + values[0] =3D LSNGetDatum(backup_state->startpoint); + values[1] =3D Int64GetDatum(backup_state->starttli); + values[2] =3D TimestampTzGetDatum(time_t_to_timestamptz(backup_state->sta= rttime)); + + /* Returns the record as Datum */ + PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)= )); } =20 =20 @@ -113,14 +126,12 @@ pg_backup_start(PG_FUNCTION_ARGS) * allows the user to choose if they want to wait for the WAL to be archiv= ed * or if we should just return as soon as the WAL record is written. * - * This function stops an in-progress backup, creates backup_label content= s and - * it returns the backup stop LSN, backup_label and tablespace_map content= s. + * This function stops an in-progress backup and returns the backup stop L= SN, + * pg_control and tablespace_map contents. * - * The backup_label contains the user-supplied label string (typically this - * would be used to tell where the backup dump will be stored), the starti= ng - * time, starting WAL location for the dump and so on. It is the caller's - * responsibility to write the backup_label and tablespace_map files in the - * data folder that will be restored from this backup. + * The pg_control file contains the recovery information for the backup. = It is + * the caller's responsibility to write the pg_control and tablespace_map = files + * in the data folder that will be restored from this backup. * * Permission checking for this function is managed through the normal * GRANT system. @@ -128,12 +139,12 @@ pg_backup_start(PG_FUNCTION_ARGS) Datum pg_backup_stop(PG_FUNCTION_ARGS) { -#define PG_BACKUP_STOP_V2_COLS 3 +#define PG_BACKUP_STOP_V2_COLS 5 TupleDesc tupdesc; Datum values[PG_BACKUP_STOP_V2_COLS] =3D {0}; bool nulls[PG_BACKUP_STOP_V2_COLS] =3D {0}; bool waitforarchive =3D PG_GETARG_BOOL(0); - char *backup_label; + bytea *pg_control_bytea; SessionBackupState status =3D get_backup_status(); =20 /* Initialize attributes information in the tuple descriptor */ @@ -152,15 +163,16 @@ pg_backup_stop(PG_FUNCTION_ARGS) /* Stop the backup */ do_pg_backup_stop(backup_state, waitforarchive); =20 - /* Build the contents of backup_label */ - backup_label =3D build_backup_content(backup_state, false); - - values[0] =3D LSNGetDatum(backup_state->stoppoint); - values[1] =3D CStringGetTextDatum(backup_label); - values[2] =3D CStringGetTextDatum(tablespace_map->data); + /* Build the contents of pg_control */ + pg_control_bytea =3D (bytea *) palloc(PG_CONTROL_FILE_SIZE + VARHDRSZ); + SET_VARSIZE(pg_control_bytea, PG_CONTROL_FILE_SIZE + VARHDRSZ); + memcpy(VARDATA(pg_control_bytea), backup_state->controlFile, PG_CONTROL_F= ILE_SIZE); =20 - /* Deallocate backup-related variables */ - pfree(backup_label); + values[0] =3D PointerGetDatum(pg_control_bytea); + values[1] =3D CStringGetTextDatum(tablespace_map->data); + values[2] =3D LSNGetDatum(backup_state->stoppoint); + values[3] =3D Int64GetDatum(backup_state->stoptli); + values[4] =3D TimestampTzGetDatum(time_t_to_timestamptz(backup_state->sto= ptime)); =20 /* Clean up the session-level state and its memory context */ backup_state =3D NULL; diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access= /transam/xlogrecovery.c index c61566666a..d05ff41f60 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -6,7 +6,7 @@ * This source file contains functions controlling WAL recovery. * InitWalRecovery() initializes the system for crash or archive recovery, * or standby mode, depending on configuration options and the state of - * the control file and possible backup label file. PerformWalRecovery() + * the control file and possible backup recovery. PerformWalRecovery() * performs the actual WAL replay, calling the rmgr-specific redo routines. * FinishWalRecovery() performs end-of-recovery checks and cleanup actions, * and prepares information needed to initialize the WAL for writes. In @@ -152,11 +152,12 @@ static bool recovery_signal_file_found =3D false; =20 /* * CheckPointLoc is the position of the checkpoint record that determines - * where to start the replay. It comes from the backup label file or the - * control file. + * where to start the replay. It comes from the control file, either from= the + * default location or from a backup recovery field. * - * RedoStartLSN is the checkpoint's REDO location, also from the backup la= bel - * file or the control file. In standby mode, XLOG streaming usually star= ts + * RedoStartLSN is the checkpoint's REDO location, also from the default + * control file location or from a backup recovery field. In standby mode, + * XLOG streaming usually starts * from the position where an invalid record was found. But if we fail to * read even the initial checkpoint record, we use the REDO location inste= ad * of the checkpoint location as the start position of XLOG streaming. @@ -388,9 +389,6 @@ static void ApplyWalRecord(XLogReaderState *xlogreader,= XLogRecord *record, Time static void EnableStandbyMode(void); static void readRecoverySignalFile(void); static void validateRecoveryParameters(void); -static bool read_backup_label(XLogRecPtr *checkPointLoc, - TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby); static bool read_tablespace_map(List **tablespaces); =20 static void xlogrecovery_redo(XLogReaderState *record, TimeLineID replayTL= I); @@ -492,8 +490,8 @@ EnableStandbyMode(void) * Prepare the system for WAL recovery, if needed. * * This is called by StartupXLOG() which coordinates the server startup - * sequence. This function analyzes the control file and the backup label - * file, if any, and figures out whether we need to perform crash recovery= or + * sequence. This function analyzes the control file and backup recovery + * info, if any, and figures out whether we need to perform crash recovery= or * archive recovery, and how far we need to replay the WAL to reach a * consistent state. * @@ -510,7 +508,7 @@ EnableStandbyMode(void) */ void InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, - bool *haveBackupLabel_ptr, bool *haveTblspcMap_ptr) + bool *haveTblspcMap_ptr) { XLogPageReadPrivate *private; struct stat st; @@ -518,7 +516,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, XLogRecord *record; DBState dbstate_at_startup; bool haveTblspcMap =3D false; - bool haveBackupLabel =3D false; + bool backupRecoveryRequired =3D false; CheckPoint checkPoint; bool backupFromStandby =3D false; =20 @@ -549,7 +547,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, =20 /* * Set the WAL reading processor now, as it will be needed when reading - * the checkpoint record required (backup_label or not). + * the checkpoint record required (backup recovery required or not). */ private =3D palloc0(sizeof(XLogPageReadPrivate)); xlogreader =3D @@ -585,18 +583,33 @@ InitWalRecovery(ControlFileData *ControlFile, bool *w= asShutdown_ptr, primary_image_masked =3D (char *) palloc(BLCKSZ); =20 /* - * Read the backup_label file. We want to run this part of the recovery - * process after checking for signal files and after performing validation - * of the recovery parameters. + * Load recovery settings from pg_control. We want to run this part of t= he + * recovery process after checking for signal files and after performing + * validation of the recovery parameters. */ - if (read_backup_label(&CheckPointLoc, &CheckPointTLI, &backupEndRequired, - &backupFromStandby)) + if (ControlFile->backupRecoveryRequired) { List *tablespaces =3D NIL; =20 + /* Initialize recovery from fields stored in pg_control */ + CheckPointLoc =3D ControlFile->backupCheckPoint; + CheckPointTLI =3D ControlFile->backupStartPointTLI; + RedoStartLSN =3D ControlFile->backupStartPoint; + RedoStartTLI =3D ControlFile->backupStartPointTLI; + backupEndRequired =3D ControlFile->backupEndRequired; + backupFromStandby =3D ControlFile->backupFromStandby; + + /* + * Clear backupRecoveryRequired in ControlFile so we do not initialize + * recovery settings again, but also set a local variable so later logic + * knows that backup recovery was initialized. + */ + ControlFile->backupRecoveryRequired =3D false; + backupRecoveryRequired =3D true; + /* - * Archive recovery was requested, and thanks to the backup label - * file, we know how far we need to replay to reach consistency. Enter + * Archive recovery was requested, and thanks to the recovery + * info, we know how far we need to replay to reach consistency. Enter * archive recovery directly. */ InArchiveRecovery =3D true; @@ -604,8 +617,9 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, EnableStandbyMode(); =20 /* - * When a backup_label file is present, we want to roll forward from - * the checkpoint it identifies, rather than using pg_control. + * When backup recovery is requested, we want to roll forward from + * the checkpoint it identifies, rather than using the default + * checkpoint. */ record =3D ReadCheckpointRecord(xlogprefetcher, CheckPointLoc, CheckPointTLI); @@ -620,9 +634,8 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, =20 /* * Make sure that REDO location exists. This may not be the case - * if there was a crash during an online backup, which left a - * backup_label around that references a WAL segment that's - * already been archived. + * if recovery.signal is missing and the WAL has already been + * archived. */ if (checkPoint.redo < CheckPointLoc) { @@ -631,20 +644,16 @@ InitWalRecovery(ControlFileData *ControlFile, bool *w= asShutdown_ptr, checkPoint.ThisTimeLineID)) ereport(FATAL, (errmsg("could not find redo location referenced by checkpoint reco= rd"), - errhint("If you are restoring from a backup, touch \"%s/recovery.s= ignal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \= "%s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corru= pt cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.s= ignal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); } } else { ereport(FATAL, (errmsg("could not locate required checkpoint record"), - errhint("If you are restoring from a backup, touch \"%s/recovery.sig= nal\" or \"%s/standby.signal\" and add required recovery options.\n" - "If you are not restoring from a backup, try removing the file \"%= s/backup_label\".\n" - "Be careful: removing \"%s/backup_label\" will result in a corrupt= cluster if restoring from a backup.", - DataDir, DataDir, DataDir, DataDir))); + errhint("If you are restoring from a backup, touch \"%s/recovery.sig= nal\" or \"%s/standby.signal\" and add required recovery options.\n", + DataDir, DataDir))); wasShutdown =3D false; /* keep compiler quiet */ } =20 @@ -679,37 +688,32 @@ InitWalRecovery(ControlFileData *ControlFile, bool *w= asShutdown_ptr, /* tell the caller to delete it later */ haveTblspcMap =3D true; } - - /* tell the caller to delete it later */ - haveBackupLabel =3D true; } else { - /* No backup_label file has been found if we are here. */ - /* - * If tablespace_map file is present without backup_label file, there - * is no use of such file. There is no harm in retaining it, but it - * is better to get rid of the map file so that we don't have any + * If tablespace_map file is present without backup recovery requested, + * there is no use of such file. There is no harm in retaining it, but + * it is better to get rid of the map file so that we don't have any * redundant file in data directory and it will avoid any sort of * confusion. It seems prudent though to just rename the file out of * the way rather than delete it completely, also we ignore any error * that occurs in rename operation as even if map file is present - * without backup_label file, it is harmless. + * without backup recovery requested, it is harmless. */ if (stat(TABLESPACE_MAP, &st) =3D=3D 0) { unlink(TABLESPACE_MAP_OLD); if (durable_rename(TABLESPACE_MAP, TABLESPACE_MAP_OLD, DEBUG1) =3D=3D 0) ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not reques= ted", + TABLESPACE_MAP), errdetail("File \"%s\" was renamed to \"%s\".", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); else ereport(LOG, - (errmsg("ignoring file \"%s\" because no file \"%s\" exists", - TABLESPACE_MAP, BACKUP_LABEL_FILE), + (errmsg("ignoring file \"%s\" because backup recovery was not reques= ted", + TABLESPACE_MAP), errdetail("Could not rename file \"%s\" to \"%s\": %m.", TABLESPACE_MAP, TABLESPACE_MAP_OLD))); } @@ -943,7 +947,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, * Any other state indicates that the backup somehow became corrupted * and we can't sensibly continue with recovery. */ - if (haveBackupLabel) + if (backupRecoveryRequired) { ControlFile->backupStartPoint =3D checkPoint.redo; ControlFile->backupEndRequired =3D backupEndRequired; @@ -953,7 +957,7 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, if (dbstate_at_startup !=3D DB_IN_ARCHIVE_RECOVERY && dbstate_at_startup !=3D DB_SHUTDOWNED_IN_RECOVERY) ereport(FATAL, - (errmsg("backup_label contains data inconsistent with control file"= ), + (errmsg("pg_control contains inconsistent data for standby backup"), errhint("This means that the backup is corrupted and you will " "have to use another backup for recovery."))); ControlFile->backupEndPoint =3D ControlFile->minRecoveryPoint; @@ -983,7 +987,6 @@ InitWalRecovery(ControlFileData *ControlFile, bool *was= Shutdown_ptr, missingContrecPtr =3D InvalidXLogRecPtr; =20 *wasShutdown_ptr =3D wasShutdown; - *haveBackupLabel_ptr =3D haveBackupLabel; *haveTblspcMap_ptr =3D haveTblspcMap; } =20 @@ -1156,154 +1159,6 @@ validateRecoveryParameters(void) } } =20 -/* - * read_backup_label: check to see if a backup_label file is present - * - * If we see a backup_label during recovery, we assume that we are recover= ing - * from a backup dump file, and we therefore roll forward from the checkpo= int - * identified by the label file, NOT what pg_control says. This avoids the - * problem that pg_control might have been archived one or more checkpoints - * later than the start of the dump, and so if we rely on it as the start - * point, we will fail to restore a consistent database state. - * - * Returns true if a backup_label was found (and fills the checkpoint - * location and TLI into *checkPointLoc and *backupLabelTLI, respectively); - * returns false if not. If this backup_label came from a streamed backup, - * *backupEndRequired is set to true. If this backup_label was created dur= ing - * recovery, *backupFromStandby is set to true. - * - * Also sets the global variables RedoStartLSN and RedoStartTLI with the L= SN - * and TLI read from the backup file. - */ -static bool -read_backup_label(XLogRecPtr *checkPointLoc, TimeLineID *backupLabelTLI, - bool *backupEndRequired, bool *backupFromStandby) -{ - char startxlogfilename[MAXFNAMELEN]; - TimeLineID tli_from_walseg, - tli_from_file; - FILE *lfp; - char ch; - char backuptype[20]; - char backupfrom[20]; - char backuplabel[MAXPGPATH]; - char backuptime[128]; - uint32 hi, - lo; - - /* suppress possible uninitialized-variable warnings */ - *checkPointLoc =3D InvalidXLogRecPtr; - *backupLabelTLI =3D 0; - *backupEndRequired =3D false; - *backupFromStandby =3D false; - - /* - * See if label file is present - */ - lfp =3D AllocateFile(BACKUP_LABEL_FILE, "r"); - if (!lfp) - { - if (errno !=3D ENOENT) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - return false; /* it's not there, all is fine */ - } - - /* - * Read and parse the START WAL LOCATION and CHECKPOINT lines (this code - * is pretty crude, but we are not expecting any variability in the file - * format). - */ - if (fscanf(lfp, "START WAL LOCATION: %X/%X (file %08X%16s)%c", - &hi, &lo, &tli_from_walseg, startxlogfilename, &ch) !=3D 5 || ch != =3D '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - RedoStartLSN =3D ((uint64) hi) << 32 | lo; - RedoStartTLI =3D tli_from_walseg; - if (fscanf(lfp, "CHECKPOINT LOCATION: %X/%X%c", - &hi, &lo, &ch) !=3D 3 || ch !=3D '\n') - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE))); - *checkPointLoc =3D ((uint64) hi) << 32 | lo; - *backupLabelTLI =3D tli_from_walseg; - - /* - * BACKUP METHOD lets us know if this was a typical backup ("streamed", - * which could mean either pg_basebackup or the pg_backup_start/stop - * method was used) or if this label came from somewhere else (the only - * other option today being from pg_rewind). If this was a streamed - * backup then we know that we need to play through until we get to the - * end of the WAL which was generated during the backup (at which point we - * will have reached consistency and backupEndRequired will be reset to be - * false). - */ - if (fscanf(lfp, "BACKUP METHOD: %19s\n", backuptype) =3D=3D 1) - { - if (strcmp(backuptype, "streamed") =3D=3D 0) - *backupEndRequired =3D true; - } - - /* - * BACKUP FROM lets us know if this was from a primary or a standby. If - * it was from a standby, we'll double-check that the control file state - * matches that of a standby. - */ - if (fscanf(lfp, "BACKUP FROM: %19s\n", backupfrom) =3D=3D 1) - { - if (strcmp(backupfrom, "standby") =3D=3D 0) - *backupFromStandby =3D true; - } - - /* - * Parse START TIME and LABEL. Those are not mandatory fields for recovery - * but checking for their presence is useful for debugging and the next - * sanity checks. Cope also with the fact that the result buffers have a - * pre-allocated size, hence if the backup_label file has been generated - * with strings longer than the maximum assumed here an incorrect parsing - * happens. That's fine as only minor consistency checks are done - * afterwards. - */ - if (fscanf(lfp, "START TIME: %127[^\n]\n", backuptime) =3D=3D 1) - ereport(DEBUG1, - (errmsg_internal("backup time %s in file \"%s\"", - backuptime, BACKUP_LABEL_FILE))); - - if (fscanf(lfp, "LABEL: %1023[^\n]\n", backuplabel) =3D=3D 1) - ereport(DEBUG1, - (errmsg_internal("backup label %s in file \"%s\"", - backuplabel, BACKUP_LABEL_FILE))); - - /* - * START TIMELINE is new as of 11. Its parsing is not mandatory, still use - * it as a sanity check if present. - */ - if (fscanf(lfp, "START TIMELINE: %u\n", &tli_from_file) =3D=3D 1) - { - if (tli_from_walseg !=3D tli_from_file) - ereport(FATAL, - (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), - errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE), - errdetail("Timeline ID parsed is %u, but expected %u.", - tli_from_file, tli_from_walseg))); - - ereport(DEBUG1, - (errmsg_internal("backup timeline %u in file \"%s\"", - tli_from_file, BACKUP_LABEL_FILE))); - } - - if (ferror(lfp) || FreeFile(lfp)) - ereport(FATAL, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", - BACKUP_LABEL_FILE))); - - return true; -} - /* * read_tablespace_map: check to see if a tablespace_map file is present * diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebacku= p.c index c7b3ba3e6e..f6cd7d954c 100644 --- a/src/backend/backup/basebackup.c +++ b/src/backend/backup/basebackup.c @@ -22,6 +22,7 @@ #include "backup/basebackup.h" #include "backup/basebackup_sink.h" #include "backup/basebackup_target.h" +#include "catalog/pg_control.h" #include "commands/defrem.h" #include "common/compression.h" #include "common/file_perm.h" @@ -192,10 +193,9 @@ static const struct exclude_list_item excludeFiles[] = =3D {RELCACHE_INIT_FILENAME, true}, =20 /* - * backup_label and tablespace_map should not exist in a running cluster - * capable of doing an online backup, but exclude them just in case. + * tablespace_map should not exist in a running cluster capable of doing + * an online backup, but exclude it just in case. */ - {BACKUP_LABEL_FILE, false}, {TABLESPACE_MAP, false}, =20 /* @@ -310,19 +310,11 @@ perform_base_backup(basebackup_options *opt, bbsink *= sink) =20 if (ti->path =3D=3D NULL) { - struct stat statbuf; bool sendtblspclinks =3D true; - char *backup_label; =20 bbsink_begin_archive(sink, "base.tar"); =20 - /* In the main tar, include the backup_label first... */ - backup_label =3D build_backup_content(backup_state, false); - sendFileWithContent(sink, BACKUP_LABEL_FILE, - backup_label, -1, &manifest); - pfree(backup_label); - - /* Then the tablespace_map file, if required... */ + /* Send the tablespace_map file, if required... */ if (opt->sendtblspcmapfile) { sendFileWithContent(sink, TABLESPACE_MAP, @@ -334,15 +326,14 @@ perform_base_backup(basebackup_options *opt, bbsink *= sink) sendDir(sink, ".", 1, false, state.tablespaces, sendtblspclinks, &manifest, InvalidOid); =20 - /* ... and pg_control after everything else. */ - if (lstat(XLOG_CONTROL_FILE, &statbuf) !=3D 0) - ereport(ERROR, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - XLOG_CONTROL_FILE))); - sendFile(sink, XLOG_CONTROL_FILE, XLOG_CONTROL_FILE, &statbuf, - false, InvalidOid, InvalidOid, - InvalidRelFileNumber, 0, &manifest); + /* End the backup before sending pg_control */ + basebackup_progress_wait_wal_archive(&state); + do_pg_backup_stop(backup_state, !opt->nowait); + + /* Send copy of pg_control containing recovery info */ + sendFileWithContent(sink, XLOG_CONTROL_FILE, + (char *)backup_state->controlFile, + PG_CONTROL_FILE_SIZE, &manifest); } else { @@ -376,9 +367,6 @@ perform_base_backup(basebackup_options *opt, bbsink *si= nk) } } =20 - basebackup_progress_wait_wal_archive(&state); - do_pg_backup_stop(backup_state, !opt->nowait); - endptr =3D backup_state->stoppoint; endtli =3D backup_state->stoptli; =20 @@ -1051,7 +1039,6 @@ sendFileWithContent(bbsink *sink, const char *filenam= e, const char *content, /* * Construct a stat struct for the file we're injecting in the tar. */ - /* Windows doesn't have the concept of uid and gid */ #ifdef WIN32 statbuf.st_uid =3D 0; diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog= /system_functions.sql index 4206752881..1f37719cf2 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -384,13 +384,15 @@ BEGIN ATOMIC END; =20 CREATE OR REPLACE FUNCTION - pg_backup_start(label text, fast boolean DEFAULT false) - RETURNS pg_lsn STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' + pg_backup_start(label text, fast boolean DEFAULT false, OUT lsn pg_lsn, + OUT timeline_id int8, OUT start timestamptz) + RETURNS record STRICT VOLATILE LANGUAGE internal AS 'pg_backup_start' PARALLEL RESTRICTED; =20 CREATE OR REPLACE FUNCTION pg_backup_stop ( - wait_for_archive boolean DEFAULT true, OUT lsn pg_lsn, - OUT labelfile text, OUT spcmapfile text) + wait_for_archive boolean DEFAULT true, OUT pg_control_file bytea, + OUT tablespace_map_file text, OUT lsn pg_lsn, OUT timeline_id int8, + OUT stop timestamptz) RETURNS record STRICT VOLATILE LANGUAGE internal as 'pg_backup_stop' PARALLEL RESTRICTED; =20 diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/mi= sc/pg_controldata.c index a1003a464d..44c4e987ce 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -163,8 +163,8 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) Datum pg_control_recovery(PG_FUNCTION_ARGS) { - Datum values[5]; - bool nulls[5]; + Datum values[9]; + bool nulls[9]; TupleDesc tupdesc; HeapTuple htup; ControlFileData *ControlFile; @@ -187,15 +187,27 @@ pg_control_recovery(PG_FUNCTION_ARGS) values[1] =3D Int32GetDatum(ControlFile->minRecoveryPointTLI); nulls[1] =3D false; =20 - values[2] =3D LSNGetDatum(ControlFile->backupStartPoint); + values[2] =3D LSNGetDatum(ControlFile->backupCheckPoint); nulls[2] =3D false; =20 - values[3] =3D LSNGetDatum(ControlFile->backupEndPoint); + values[3] =3D LSNGetDatum(ControlFile->backupStartPoint); nulls[3] =3D false; =20 - values[4] =3D BoolGetDatum(ControlFile->backupEndRequired); + values[4] =3D Int32GetDatum(ControlFile->backupStartPointTLI); nulls[4] =3D false; =20 + values[5] =3D LSNGetDatum(ControlFile->backupEndPoint); + nulls[5] =3D false; + + values[6] =3D BoolGetDatum(ControlFile->backupRecoveryRequired); + nulls[6] =3D false; + + values[7] =3D BoolGetDatum(ControlFile->backupFromStandby); + nulls[7] =3D false; + + values[8] =3D BoolGetDatum(ControlFile->backupEndRequired); + nulls[8] =3D false; + htup =3D heap_form_tuple(tupdesc, values, nulls); =20 PG_RETURN_DATUM(HeapTupleGetDatum(htup)); diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_base= backup/t/010_pg_basebackup.pl index b9f5e1266b..c655cb0335 100644 --- a/src/bin/pg_basebackup/t/010_pg_basebackup.pl +++ b/src/bin/pg_basebackup/t/010_pg_basebackup.pl @@ -171,8 +171,8 @@ SKIP: =20 # Write some files to test that they are not copied. foreach my $filename ( - qw(backup_label tablespace_map postgresql.auto.conf.tmp - current_logfiles.tmp global/pg_internal.init.123)) + qw(tablespace_map postgresql.auto.conf.tmp current_logfiles.tmp + global/pg_internal.init.123)) { open my $file, '>>', "$pgdata/$filename"; print $file "DONOTCOPY"; @@ -261,14 +261,13 @@ foreach my $filename (@tempRelationFiles) "base/$postgresOid/$filename not copied"); } =20 -# Make sure existing backup_label was ignored. -isnt(slurp_file("$tempdir/backup/backup_label"), - 'DONOTCOPY', 'existing backup_label not copied'); +# Make sure existing tablespace_map was ignored. +ok(!-f "$tempdir/backup/tablespace_map", 'tablespace_map not in backup'); rmtree("$tempdir/backup"); =20 -# Now delete the bogus backup_label file since it will interfere with star= tup -unlink("$pgdata/backup_label") - or BAIL_OUT("unable to unlink $pgdata/backup_label"); +# Now delete the bogus tablespace_map file since it will interfere with st= artup +unlink("$pgdata/tablespace_map") + or BAIL_OUT("unable to unlink $pgdata/tablespace_map"); =20 $node->command_ok( [ diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controlda= ta/pg_controldata.c index 93e0837947..cc515b622f 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -277,10 +277,18 @@ main(int argc, char *argv[]) LSN_FORMAT_ARGS(ControlFile->minRecoveryPoint)); printf(_("Min recovery ending loc's timeline: %u\n"), ControlFile->minRecoveryPointTLI); + printf(_("Backup checkpoint location: %X/%X\n"), + LSN_FORMAT_ARGS(ControlFile->backupCheckPoint)); printf(_("Backup start location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupStartPoint)); + printf(_("Backup start location's timeline: %u\n"), + ControlFile->backupStartPointTLI); printf(_("Backup end location: %X/%X\n"), LSN_FORMAT_ARGS(ControlFile->backupEndPoint)); + printf(_("Backup recovery required: %s\n"), + ControlFile->backupRecoveryRequired ? _("yes") : _("no")); + printf(_("Backup from standby: %s\n"), + ControlFile->backupFromStandby ? _("yes") : _("no")); printf(_("End-of-backup record required: %s\n"), ControlFile->backupEndRequired ? _("yes") : _("no")); printf(_("wal_level setting: %s\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_res= etwal.c index 3ae3fc06df..255101ff3a 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -870,8 +870,12 @@ RewriteControlFile(void) ControlFile.checkPoint =3D ControlFile.checkPointCopy.redo; ControlFile.minRecoveryPoint =3D 0; ControlFile.minRecoveryPointTLI =3D 0; + ControlFile.backupCheckPoint =3D 0; ControlFile.backupStartPoint =3D 0; + ControlFile.backupStartPointTLI =3D 0; ControlFile.backupEndPoint =3D 0; + ControlFile.backupRecoveryRequired =3D false; + ControlFile.backupFromStandby =3D false; ControlFile.backupEndRequired =3D false; =20 /* diff --git a/src/bin/pg_rewind/filemap.c b/src/bin/pg_rewind/filemap.c index ecadd69dc5..213f4e71b8 100644 --- a/src/bin/pg_rewind/filemap.c +++ b/src/bin/pg_rewind/filemap.c @@ -139,11 +139,10 @@ static const struct exclude_list_item excludeFiles[] = =3D {"pg_internal.init", true}, /* defined as RELCACHE_INIT_FILENAME */ =20 /* - * If there is a backup_label or tablespace_map file, it indicates that a - * recovery failed and this cluster probably can't be rewound, but exclude - * them anyway if they are found. + * If there is a tablespace_map file, it indicates that a recovery failed + * and this cluster probably can't be rewound, but exclude it anyway if it + * is found. */ - {"backup_label", false}, /* defined as BACKUP_LABEL_FILE */ {"tablespace_map", false}, /* defined as TABLESPACE_MAP */ =20 /* diff --git a/src/bin/pg_rewind/pg_rewind.c b/src/bin/pg_rewind/pg_rewind.c index bfd44a284e..6b23ed4a1c 100644 --- a/src/bin/pg_rewind/pg_rewind.c +++ b/src/bin/pg_rewind/pg_rewind.c @@ -39,9 +39,6 @@ static void perform_rewind(filemap_t *filemap, rewind_sou= rce *source, TimeLineID chkpttli, XLogRecPtr chkptredo); =20 -static void createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, - XLogRecPtr checkpointloc); - static void digestControlFile(ControlFileData *ControlFile, const char *content, size_t size); static void getRestoreCommand(const char *argv0); @@ -651,10 +648,10 @@ perform_rewind(filemap_t *filemap, rewind_source *sou= rce, } =20 if (showprogress) - pg_log_info("creating backup label and updating control file"); + pg_log_info("updating control file"); =20 /* - * Create a backup label file, to tell the target where to begin the WAL + * Get recovery fields to tell the target where to begin the WAL * replay. Normally, from the last common checkpoint between the source * and the target. But if the source is a standby server, it's possible * that the last common checkpoint is *after* the standby's restartpoint. @@ -672,7 +669,6 @@ perform_rewind(filemap_t *filemap, rewind_source *sourc= e, chkpttli =3D ControlFile_source.checkPointCopy.ThisTimeLineID; chkptrec =3D ControlFile_source.checkPoint; } - createBackupLabel(chkptredo, chkpttli, chkptrec); =20 /* * Update control file of target, to tell the target how far it must @@ -722,6 +718,12 @@ perform_rewind(filemap_t *filemap, rewind_source *sour= ce, ControlFile_new.minRecoveryPoint =3D endrec; ControlFile_new.minRecoveryPointTLI =3D endtli; ControlFile_new.state =3D DB_IN_ARCHIVE_RECOVERY; + ControlFile_new.backupRecoveryRequired =3D true; + ControlFile_new.backupFromStandby =3D true; + ControlFile_new.backupEndRequired =3D false; + ControlFile_new.backupCheckPoint =3D chkptrec; + ControlFile_new.backupStartPoint =3D chkptredo; + ControlFile_new.backupStartPointTLI =3D chkpttli; if (!dry_run) update_controlfile(datadir_target, &ControlFile_new, do_sync); } @@ -729,7 +731,10 @@ perform_rewind(filemap_t *filemap, rewind_source *sour= ce, static void sanityChecks(void) { - /* TODO Check that there's no backup_label in either cluster */ + /* + * TODO Check that neither cluster has backupRecoveryRequested set in + * pg_control. + */ =20 /* Check system_identifier match */ if (ControlFile_target.system_identifier !=3D ControlFile_source.system_i= dentifier) @@ -951,51 +956,6 @@ findCommonAncestorTimeline(TimeLineHistoryEntry *a_his= tory, int a_nentries, } } =20 - -/* - * Create a backup_label file that forces recovery to begin at the last co= mmon - * checkpoint. - */ -static void -createBackupLabel(XLogRecPtr startpoint, TimeLineID starttli, XLogRecPtr c= heckpointloc) -{ - XLogSegNo startsegno; - time_t stamp_time; - char strfbuf[128]; - char xlogfilename[MAXFNAMELEN]; - struct tm *tmp; - char buf[1000]; - int len; - - XLByteToSeg(startpoint, startsegno, WalSegSz); - XLogFileName(xlogfilename, starttli, startsegno, WalSegSz); - - /* - * Construct backup label file - */ - stamp_time =3D time(NULL); - tmp =3D localtime(&stamp_time); - strftime(strfbuf, sizeof(strfbuf), "%Y-%m-%d %H:%M:%S %Z", tmp); - - len =3D snprintf(buf, sizeof(buf), - "START WAL LOCATION: %X/%X (file %s)\n" - "CHECKPOINT LOCATION: %X/%X\n" - "BACKUP METHOD: pg_rewind\n" - "BACKUP FROM: standby\n" - "START TIME: %s\n", - /* omit LABEL: line */ - LSN_FORMAT_ARGS(startpoint), xlogfilename, - LSN_FORMAT_ARGS(checkpointloc), - strfbuf); - if (len >=3D sizeof(buf)) - pg_fatal("backup label buffer too small"); /* shouldn't happen */ - - /* TODO: move old file out of the way, if any. */ - open_target_file("backup_label", true); /* BACKUP_LABEL_FILE */ - write_target_range(buf, 0, len); - close_target_file(); -} - /* * Check CRC of control file */ diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index a14126d164..3aac6839a7 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -293,8 +293,6 @@ extern SessionBackupState get_backup_status(void); /* File path names (all relative to $PGDATA) */ #define RECOVERY_SIGNAL_FILE "recovery.signal" #define STANDBY_SIGNAL_FILE "standby.signal" -#define BACKUP_LABEL_FILE "backup_label" -#define BACKUP_LABEL_OLD "backup_label.old" =20 #define TABLESPACE_MAP "tablespace_map" #define TABLESPACE_MAP_OLD "tablespace_map.old" diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbacku= p.h index 1611358137..9be80a4f7d 100644 --- a/src/include/access/xlogbackup.h +++ b/src/include/access/xlogbackup.h @@ -15,6 +15,7 @@ #define XLOG_BACKUP_H =20 #include "access/xlogdefs.h" +#include "catalog/pg_control.h" #include "pgtime.h" =20 /* Structure to hold backup state. */ @@ -33,9 +34,16 @@ typedef struct BackupState XLogRecPtr stoppoint; /* backup stop WAL location */ TimeLineID stoptli; /* backup stop TLI */ pg_time_t stoptime; /* backup stop time */ + + /* + * After pg_backup_stop() returns this field will contain a copy of + * pg_control that should be stored with the backup. Fields have been + * updated for recovery and the CRC has been recalculated. Bytes after + * sizeof(ControlFileData) are zeroed. + */ + uint8_t controlFile[PG_CONTROL_FILE_SIZE]; } BackupState; =20 -extern char *build_backup_content(BackupState *state, - bool ishistoryfile); +extern char *build_backup_history_content(BackupState *state); =20 #endif /* XLOG_BACKUP_H */ diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrec= overy.h index ee0bc74278..981266f734 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -80,8 +80,7 @@ extern Size XLogRecoveryShmemSize(void); extern void XLogRecoveryShmemInit(void); =20 extern void InitWalRecovery(ControlFileData *ControlFile, - bool *wasShutdown_ptr, bool *haveBackupLabel_ptr, - bool *haveTblspcMap_ptr); + bool *wasShutdown_ptr, bool *haveTblspcMap_ptr); extern void PerformWalRecovery(void); =20 /* diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_cont= rol.h index 2ae72e3b26..0ea2e73368 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -146,6 +146,9 @@ typedef struct ControlFileData * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * + * backupCheckPoint is the backup start checkpoint and is set to zero aft= er + * recovery is initialized. + * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and @@ -160,14 +163,28 @@ typedef struct ControlFileData * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * + * backupRecoveryRequired indicates that the pg_control file was provided + * by a backup or pg_rewind and recovery settings need to be copied to the + * appropriate fields. It will be set to false when the settings have been + * copied. + * + * backupFromStandby indicates that the backup was taken on a standby. It= is + * required to initialize recovery and set to false afterwards. + * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely - * start up. + * start up. Currently backupEndRequired should only be false if recovery + * settings were configured by pg_rewind, which does not require an end + * point. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; + XLogRecPtr backupCheckPoint; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; + TimeLineID backupStartPointTLI; + bool backupRecoveryRequired; + bool backupFromStandby; bool backupEndRequired; =20 /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.= dat index fb58dee3bc..16346f0540 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6427,13 +6427,17 @@ prosrc =3D> 'pg_terminate_backend' }, { oid =3D> '2172', descr =3D> 'prepare for taking an online backup', proname =3D> 'pg_backup_start', provolatile =3D> 'v', proparallel =3D> '= r', - prorettype =3D> 'pg_lsn', proargtypes =3D> 'text bool', + prorettype =3D> 'record', proargtypes =3D> 'text bool', + proallargtypes =3D> '{text,bool,pg_lsn,int8,timestamptz}', + proargmodes =3D> '{i,i,o,o,o}', + proargnames =3D> '{label,fast,lsn,timeline_id,start}', prosrc =3D> 'pg_backup_start' }, { oid =3D> '2739', descr =3D> 'finish taking an online backup', proname =3D> 'pg_backup_stop', provolatile =3D> 'v', proparallel =3D> 'r= ', prorettype =3D> 'record', proargtypes =3D> 'bool', - proallargtypes =3D> '{bool,pg_lsn,text,text}', proargmodes =3D> '{i,o,o,= o}', - proargnames =3D> '{wait_for_archive,lsn,labelfile,spcmapfile}', + proallargtypes =3D> '{bool,bytea,text,pg_lsn,int8,timestamptz}', + proargmodes =3D> '{i,o,o,o,o,o}', + proargnames =3D> '{wait_for_archive,pg_control_file,tablespace_map_file,= lsn,timeline_id,stop}', prosrc =3D> 'pg_backup_stop' }, { oid =3D> '3436', descr =3D> 'promote standby server', proname =3D> 'pg_promote', provolatile =3D> 'v', prorettype =3D> 'bool', @@ -11917,9 +11921,9 @@ { oid =3D> '3443', descr =3D> 'pg_controldata recovery state information as a function', proname =3D> 'pg_control_recovery', provolatile =3D> 'v', prorettype =3D= > 'record', - proargtypes =3D> '', proallargtypes =3D> '{pg_lsn,int4,pg_lsn,pg_lsn,boo= l}', - proargmodes =3D> '{o,o,o,o,o}', - proargnames =3D> '{min_recovery_end_lsn,min_recovery_end_timeline,backup= _start_lsn,backup_end_lsn,end_of_backup_record_required}', + proargtypes =3D> '', proallargtypes =3D> '{pg_lsn,int4,pg_lsn,pg_lsn,int= 4,pg_lsn,bool,bool,bool}', + proargmodes =3D> '{o,o,o,o,o,o,o,o,o}', + proargnames =3D> '{min_recovery_end_lsn,min_recovery_end_timeline,backup= _checkpoint_lsn,backup_start_lsn,backup_start_tli,backup_end_lsn,backup_rec= overy_required,backup_from_standby,end_of_backup_record_required}', prosrc =3D> 'pg_control_recovery' }, =20 { oid =3D> '3444', --=20 2.34.1 --W0+6J45DoLYiQ+WQ-- --jTQITlajykV8t6gp Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmVas0QACgkQnvQgOdby QH3YjhAAgRBu6d0foOAuN9NjLpDzSwDs1/r0qONb9XDhXnzqChP1NogcKX7pYl6u K9CJRvLhKw2us4M11VNTMAzfNu8gAu/5Sqi0Sdjg9d8af2teFO78P3jpp0G3azEz 1Jdg5rtAnQdAsmuCZyVgqyxabHHtFNJa143p/WOf5dOKCUN8KnQTxlOfreRUK+TK H36xG9poVex1v/kbDVio9Oql2GZ/Yqhf0i/Mv6SB+wm9/qENufqoRriAqcH6cyVN mjfRPThl6WOP/eWRlSLpIjl7GpwQZuGMdihh6VRsgHiqH9UKa4aitXVYDLRnShwq SGh2ue0MX/539bnhhCap9g2ETNiHv6IzYNVzz5HFuGUOQeoySVqi17SYa77Ny53q +xtUr2f/SEgX02wkJdSm6QjTXN1pxsNpEHB2nEXp2WNJIOv03tkqk9KTRwCDLTUf YpyuJjpedEGZTqeBAB6bhedXJDDjy6dn4G+xel2YqlWBc6XKWcdnW22tvZdUlemp iEjgp8dIKiMHVyhoiyDXK30vFFriMajBsOcSvM3MIbMzV2nAoyb3UiBhZkuGkoqi WVKuFW+iY8fccRxOqlg75Cixi8LmIZzUMxc+rST0RYA63k4AqGMxESAqabb4Q73B qpy/40at3pqYswKnSJP+c+r7FxYPRzX/zIRCnHCC/ekskw899RM= =Yd45 -----END PGP SIGNATURE----- --jTQITlajykV8t6gp--