Date: Mon, 20 Apr 2020 16:34:44 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: jgdr@dalibo.com, masao.fujii@oss.nttdata.com,
	pgsql-bugs@lists.postgresql.org
Subject: Re: [BUG] non archived WAL removed during production crash recovery
Message-ID: <20200420073444.GA77439@paquier.xyz>
References: <20200417065043.GC350229@paquier.xyz>
 <20200417153304.01a226cc@firost>
 <20200418092611.GD350229@paquier.xyz>
 <20200420.160231.719176619682386260.horikyota.ntt@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="NDin8bjvE/0mNLFQ"
Content-Disposition: inline
In-Reply-To: <20200420.160231.719176619682386260.horikyota.ntt@gmail.com>
Precedence: bulk


--NDin8bjvE/0mNLFQ
Content-Type: multipart/mixed; boundary="4Ckj6UjgE2iN1+kY"
Content-Disposition: inline


--4Ckj6UjgE2iN1+kY
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Apr 20, 2020 at 04:02:31PM +0900, Kyotaro Horiguchi wrote:
> At Sat, 18 Apr 2020 18:26:11 +0900, Michael Paquier <michael@paquier.xyz>=
 wrote in=20
> As the result, +1 to what v7 is doing and discussing on earlier
> removal of such WAL segments separately if needed.

Thanks for the extra review.

>> Yeah.  We could try to do with "false" as command anyway, and see what
>> the buildfarm thinks.  As the test is skipped on Windows, I would
>> assume that it does not matter much anyway.  Let's see what others
>> think about this piece.  I don't have plans to touch again this patch
>> until likely the middle of next week.
>=20
> Couldn't we use "/" as a globally-results-in-failure command?  But
> that doesn't increment failed_count.  The reason is pgarch_archiveXLog
> exits with FATAL for "is a directory" error.  The comment asserts that
> we exit with FATAL for SIGINT or SIGQUIT and if so it is enough to
> check only exit-by-signal case.  The following fix worked.

Yeah, I was working on this stuff today and I noticed this problem.  I
was just going to send an email on the matter with a more portable
patch and you also just beat me to it with this one :)

So yes, using "false" may be a bad idea because we cannot rely on the
case where the command does not exist in an environment in this test.
After more testing, I have been hit hard about the fact that the
archiver exits immediately if an archive command cannot be found
(errcode =3D 127), and it does not report this failure back to
pg_stat_archiver, which would cause the test to wait until the timeout
of poll_query_until() kills the test.  There is however an extra
method not mentioned yet on this thread: we know that cp/copy is
portable enough per the buildfarm, so let's use a copy command that we
know *will* fail.  A simple way of doing this is a command where the
origin file does not exist.

> --- a/src/backend/postmaster/pgarch.c
> +++ b/src/backend/postmaster/pgarch.c
> @@ -595,7 +595,7 @@ pgarch_archiveXlog(char *xlog)
>  		 * "command not found" type of error.  If we overreact it's no big
>  		 * deal, the postmaster will just start the archiver again.
>  		 */
> -		int			lev =3D wait_result_is_any_signal(rc, true) ? FATAL : LOG;
> +		int			lev =3D wait_result_is_any_signal(rc, false) ? FATAL : LOG;
> =20
>  		if (WIFEXITED(rc))
>  		{
>=20
> I didn't tested it on Windows (I somehow broke my repo and it's too
> slow to clone.) but system("/") returned 1 and I think that result
> increments the counter.

No, this would be a behavior change, which is not acceptable in my
view.  (By the way, just nuke your full repo if it does not work
anymore on Windows, this method works).

>> Indeed.  The extra initialization was part of v4, and got removed as
>> of v5.  Still, it seems to me that this part was not complete without
>> updating the shared memory field correctly at the beginning of the
>> REDO processing as the last version of the patch does.
>=20
> I may not be following the discussion, but I think it is reasonable
> that SharedRecoveryState is initialized as CRASH then moves to ARCHIVE
> as needed and finished by NONE.  That transition also stables
> RecoveryInProgress().

Thought as well about that over the weekend, and that's still the best
option to me.

> I think it would be better be RECOVERY_STATE_DONE.

I like this suggestion better than the original in v7.

> By the way I noticed that RecoveryState is exactly a subset of
> DBState.  And changes of SharedRecoveryState happens side-by-side with
> ControlFileData->state in most places.  Coundn't we just usee
> ControlFile->state instead of SharedRecoveryState?

I actually found confusing to use the same thing, because then the
reader would thing that SharedRecoveryState could be set to more
values but we don't want that.

> By the way I found a typo.
>=20
> +# Recovery tests for the achiving with a standby partially check
> s/achiving/archiving/

Thanks, fixed.

Attached is an updated patch, where I tweaked more comments.

Jehan-Guillaume, who is your colleague who found originally about this
problem?  We should credit him in the commit message.
--
Michael

--4Ckj6UjgE2iN1+kY
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
	filename="v8-0001-Fix-handling-of-.ready-files-during-crash-recover.patch"
Content-Transfer-Encoding: quoted-printable

=46rom cbe9310cc2f2e065708e7e9e05cf6a94043d484d Mon Sep 17 00:00:00 2001
=46rom: Michael Paquier <michael@paquier.xyz>
Date: Mon, 20 Apr 2020 16:30:32 +0900
Subject: [PATCH v8] Fix handling of .ready files during crash recovery

78ea8b5 introduced a fix to correctly recycle WAL segments on a standby
when archive_mode =3D always, however it has introduced a regression with
the handling of .ready files during crash recovery of a primary, causing
those files to be removed.

A set of TAP tests is added to close the gap here, making sure that
archive status files are correctly handled when a cluster is in archive
or crash recovery with archive_mode set to "on" or "always, for both
standbys and primarys.

Reported-by: ???
Author: Jehan-Guillaume de Rorthais
Reviewed-by: Kyotaro Horiguchi, Fujii Masao, Michael Paquier
Backpatch-through: 9.5
---
 src/include/access/xlog.h                 |   9 +
 src/backend/access/transam/xlog.c         |  57 +++++-
 src/backend/access/transam/xlogarchive.c  |  21 +-
 src/test/recovery/t/020_archive_status.pl | 225 ++++++++++++++++++++++
 src/tools/pgindent/typedefs.list          |   1 +
 5 files changed, 297 insertions(+), 16 deletions(-)
 create mode 100644 src/test/recovery/t/020_archive_status.pl

diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f60ed2d36c..0a12afb59e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -166,6 +166,14 @@ typedef enum WalLevel
 	WAL_LEVEL_LOGICAL
 } WalLevel;
=20
+/* Recovery states */
+typedef enum RecoveryState
+{
+	RECOVERY_STATE_CRASH =3D 0,	/* crash recovery */
+	RECOVERY_STATE_ARCHIVE,		/* archive recovery */
+	RECOVERY_STATE_DONE			/* currently in production */
+} RecoveryState;
+
 extern PGDLLIMPORT int wal_level;
=20
 /* Is WAL archiving enabled (always or only while server is running normal=
ly)? */
@@ -291,6 +299,7 @@ extern const char *xlog_identify(uint8 info);
 extern void issue_xlog_fsync(int fd, XLogSegNo segno);
=20
 extern bool RecoveryInProgress(void);
+extern RecoveryState GetRecoveryState(void);
 extern bool HotStandbyActive(void);
 extern bool HotStandbyActiveInReplay(void);
 extern bool XLogInsertAllowed(void);
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam=
/xlog.c
index 11e32733c4..009a071276 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -221,8 +221,9 @@ static TimeLineID receiveTLI =3D 0;
 static bool lastFullPageWrites;
=20
 /*
- * Local copy of SharedRecoveryInProgress variable. True actually means "n=
ot
- * known, need to check the shared state".
+ * Local copy of the state tracked by SharedRecoveryState in shared memory,
+ * It is false if SharedRecoveryState is RECOVERY_STATE_DONE.  True actual=
ly
+ * means "not known, need to check the shared state".
  */
 static bool LocalRecoveryInProgress =3D true;
=20
@@ -653,10 +654,10 @@ typedef struct XLogCtlData
 	TimeLineID	PrevTimeLineID;
=20
 	/*
-	 * SharedRecoveryInProgress indicates if we're still in crash or archive
+	 * SharedRecoveryState indicates if we're still in crash or archive
 	 * recovery.  Protected by info_lck.
 	 */
-	bool		SharedRecoveryInProgress;
+	RecoveryState SharedRecoveryState;
=20
 	/*
 	 * SharedHotStandbyActive indicates if we allow hot standby queries to be
@@ -4434,6 +4435,16 @@ ReadRecord(XLogReaderState *xlogreader, int emode,
 				updateMinRecoveryPoint =3D true;
=20
 				UpdateControlFile();
+
+				/*
+				 * We update SharedRecoveryState while holding the lock on
+				 * ControlFileLock so both states are consistent in shared
+				 * memory.
+				 */
+				SpinLockAcquire(&XLogCtl->info_lck);
+				XLogCtl->SharedRecoveryState =3D RECOVERY_STATE_ARCHIVE;
+				SpinLockRelease(&XLogCtl->info_lck);
+
 				LWLockRelease(ControlFileLock);
=20
 				CheckRecoveryConsistency();
@@ -5166,7 +5177,7 @@ XLOGShmemInit(void)
 	 * in additional info.)
 	 */
 	XLogCtl->XLogCacheBlck =3D XLOGbuffers - 1;
-	XLogCtl->SharedRecoveryInProgress =3D true;
+	XLogCtl->SharedRecoveryState =3D RECOVERY_STATE_CRASH;
 	XLogCtl->SharedHotStandbyActive =3D false;
 	XLogCtl->SharedPromoteIsTriggered =3D false;
 	XLogCtl->WalWriterSleeping =3D false;
@@ -6871,7 +6882,13 @@ StartupXLOG(void)
 		 */
 		dbstate_at_startup =3D ControlFile->state;
 		if (InArchiveRecovery)
+		{
 			ControlFile->state =3D DB_IN_ARCHIVE_RECOVERY;
+
+			SpinLockAcquire(&XLogCtl->info_lck);
+			XLogCtl->SharedRecoveryState =3D RECOVERY_STATE_ARCHIVE;
+			SpinLockRelease(&XLogCtl->info_lck);
+		}
 		else
 		{
 			ereport(LOG,
@@ -6884,6 +6901,10 @@ StartupXLOG(void)
 								ControlFile->checkPointCopy.ThisTimeLineID,
 								recoveryTargetTLI)));
 			ControlFile->state =3D DB_IN_CRASH_RECOVERY;
+
+			SpinLockAcquire(&XLogCtl->info_lck);
+			XLogCtl->SharedRecoveryState =3D RECOVERY_STATE_CRASH;
+			SpinLockRelease(&XLogCtl->info_lck);
 		}
 		ControlFile->checkPoint =3D checkPointLoc;
 		ControlFile->checkPointCopy =3D checkPoint;
@@ -7911,7 +7932,7 @@ StartupXLOG(void)
 	ControlFile->time =3D (pg_time_t) time(NULL);
=20
 	SpinLockAcquire(&XLogCtl->info_lck);
-	XLogCtl->SharedRecoveryInProgress =3D false;
+	XLogCtl->SharedRecoveryState =3D RECOVERY_STATE_DONE;
 	SpinLockRelease(&XLogCtl->info_lck);
=20
 	UpdateControlFile();
@@ -8057,7 +8078,7 @@ RecoveryInProgress(void)
 		 */
 		volatile XLogCtlData *xlogctl =3D XLogCtl;
=20
-		LocalRecoveryInProgress =3D xlogctl->SharedRecoveryInProgress;
+		LocalRecoveryInProgress =3D (xlogctl->SharedRecoveryState !=3D RECOVERY_=
STATE_DONE);
=20
 		/*
 		 * Initialize TimeLineID and RedoRecPtr when we discover that recovery
@@ -8069,8 +8090,8 @@ RecoveryInProgress(void)
 		{
 			/*
 			 * If we just exited recovery, make sure we read TimeLineID and
-			 * RedoRecPtr after SharedRecoveryInProgress (for machines with
-			 * weak memory ordering).
+			 * RedoRecPtr after SharedRecoveryState (for machines with weak
+			 * memory ordering).
 			 */
 			pg_memory_barrier();
 			InitXLOGAccess();
@@ -8086,6 +8107,24 @@ RecoveryInProgress(void)
 	}
 }
=20
+/*
+ * Returns current recovery state from shared memory.
+ *
+ * This returned state is kept consistent with the contents of the control
+ * file.  See details about the possible values of RecoveryState in xlog.h.
+ */
+RecoveryState
+GetRecoveryState(void)
+{
+	RecoveryState retval;
+
+	SpinLockAcquire(&XLogCtl->info_lck);
+	retval =3D XLogCtl->SharedRecoveryState;
+	SpinLockRelease(&XLogCtl->info_lck);
+
+	return retval;
+}
+
 /*
  * Is HotStandby active yet? This is only important in special backends
  * since normal backends won't ever be able to connect until this returns
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/=
transam/xlogarchive.c
index d62c12310a..55becd65d4 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -572,18 +572,25 @@ XLogArchiveCheckDone(const char *xlog)
 {
 	char		archiveStatusPath[MAXPGPATH];
 	struct stat stat_buf;
-	bool		inRecovery =3D RecoveryInProgress();
+
+	/* The file is always deletable if archive_mode is "off". */
+	if (!XLogArchivingActive())
+		return true;
=20
 	/*
-	 * The file is always deletable if archive_mode is "off".  On standbys
-	 * archiving is disabled if archive_mode is "on", and enabled with
-	 * "always".  On a primary, archiving is enabled if archive_mode is "on"
-	 * or "always".
+	 * During archive recovery, the file is deletable if archive_mode is not
+	 * "always".
 	 */
-	if (!((XLogArchivingActive() && !inRecovery) ||
-		  (XLogArchivingAlways() && inRecovery)))
+	if (!XLogArchivingAlways() &&
+		GetRecoveryState() =3D=3D RECOVERY_STATE_ARCHIVE)
 		return true;
=20
+	/*
+	 * At this point of the logic, note that we are either a primary with
+	 * archive_mode set to "on" or "always", or a standby with archive_mode
+	 * set to "always".
+	 */
+
 	/* First check for .done --- this means archiver is done with it */
 	StatusFilePath(archiveStatusPath, xlog, ".done");
 	if (stat(archiveStatusPath, &stat_buf) =3D=3D 0)
diff --git a/src/test/recovery/t/020_archive_status.pl b/src/test/recovery/=
t/020_archive_status.pl
new file mode 100644
index 0000000000..688014862a
--- /dev/null
+++ b/src/test/recovery/t/020_archive_status.pl
@@ -0,0 +1,225 @@
+#
+# Tests related to WAL archiving and recovery.
+#
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+use Config;
+
+if ($Config{osname} eq 'MSWin32')
+{
+
+	# some Windows Perls at least don't like IPC::Run's start/kill_kill regim=
e.
+	plan skip_all =3D> "Test fails on Windows perl";
+}
+else
+{
+	plan tests =3D> 16;
+}
+
+my $primary =3D get_new_node('master');
+$primary->init(
+	has_archiving    =3D> 1,
+	allows_streaming =3D> 1);
+$primary->append_conf('postgresql.conf', 'autovacuum =3D off');
+$primary->start;
+my $primary_data =3D $primary->data_dir;
+
+# Temporarily use an archive_command value to make the archiver fail,
+# knowing that archiving is enabled.  Note that we cannot use a command
+# that does not exist as in this case the archiver process would just exit
+# without reporting the failure to pg_stat_archiver.  This also cannot
+# use a plain "false" as that's unportable on Windows.  So, instead, as
+# a portable solution, use an archive command that is known to work but
+# will fail: a copy with an incorrect original path.
+my $incorrect_command =3D
+  $TestLib::windows_os
+  ? qq{copy "%p_does_not_exist" "%f_does_not_exist"}
+  : qq{cp "%p_does_not_exist" "%f_does_not_exist"};
+$primary->safe_psql(
+	'postgres', qq{
+    ALTER SYSTEM SET archive_command TO '$incorrect_command';
+    SELECT pg_reload_conf();
+});
+
+# Save the WAL segment currently in use and switch to a new segment.
+# This will be used to track the activity of the archiver.
+my $segment_name_1 =3D $primary->safe_psql('postgres',
+	q{SELECT pg_walfile_name(pg_current_wal_lsn())});
+my $segment_path_1       =3D "pg_wal/archive_status/$segment_name_1";
+my $segment_path_1_ready =3D "$segment_path_1.ready";
+my $segment_path_1_done  =3D "$segment_path_1.done";
+$primary->safe_psql(
+	'postgres', q{
+	CREATE TABLE mine AS SELECT generate_series(1,10) AS x;
+	SELECT pg_switch_wal();
+	CHECKPOINT;
+});
+
+# Wait for an archive failure.
+$primary->poll_query_until('postgres',
+	q{SELECT failed_count > 0 FROM pg_stat_archiver}, 't')
+  or die "Timed out while waiting for archiving to fail";
+ok( -f "$primary_data/$segment_path_1_ready",
+	".ready file exists for WAL segment $segment_name_1 waiting to be archive=
d"
+);
+ok( !-f "$primary_data/$segment_path_1_done",
+	".done file does not exist for WAL segment $segment_name_1 waiting to be =
archived"
+);
+
+is( $primary->safe_psql(
+		'postgres', q{
+		SELECT archived_count, last_failed_wal
+		FROM pg_stat_archiver
+	}),
+	"0|$segment_name_1",
+	'pg_stat_archiver failed to archive $segment_name_1');
+
+# We need to crash the cluster because next test checks the crash
+# recovery step do not removes non-archived WAL.
+$primary->stop('immediate');
+
+# Recovery tests for the archiving with a standby partially check
+# the recovery behavior when restoring a backup taken using a
+# snapshot with no pg_start/stop_backup.  In this situation,
+# the recovered standby should enter first crash recovery then
+# switch to regular archive recovery.  Note that the base backup
+# is taken here so as archive_command will fail.  This is necessary
+# for the assumptions of the tests done with the standbys below.
+$primary->backup_fs_cold('backup');
+
+$primary->start;
+ok( -f "$primary_data/$segment_path_1_ready",
+	".ready file for WAL segment $segment_name_1 still exists after crash rec=
overy on primary"
+);
+
+# Allow WAL archiving again and wait for a success.
+$primary->safe_psql(
+	'postgres', q{
+	ALTER SYSTEM RESET archive_command;
+	SELECT pg_reload_conf();
+});
+
+$primary->poll_query_until('postgres',
+	q{SELECT archived_count FROM pg_stat_archiver}, '1')
+  or die "Timed out while waiting for archiving to finish";
+
+ok(!-f "$primary_data/$segment_path_1_ready",
+	".ready file for archived WAL segment $segment_name_1 removed");
+
+ok(-f "$primary_data/$segment_path_1_done",
+	".done file for archived WAL segment $segment_name_1 exists");
+
+is( $primary->safe_psql(
+		'postgres', q{ SELECT last_archived_wal FROM pg_stat_archiver }),
+	$segment_name_1,
+	'archive success reported in pg_stat_archiver for WAL segment $segment_na=
me_1'
+);
+
+# Create some WAL activity and a new checkpoint so as the next standby can
+# create a restartpoint.  As this standby starts in crash recovery because
+# of the cold backup taken previously, it needs a clean restartpoint to de=
al
+# with existing status files.
+my $segment_name_2 =3D $primary->safe_psql('postgres',
+	q{SELECT pg_walfile_name(pg_current_wal_lsn())});
+my $segment_path_2       =3D "pg_wal/archive_status/$segment_name_2";
+my $segment_path_2_ready =3D "$segment_path_2.ready";
+my $segment_path_2_done  =3D "$segment_path_2.done";
+$primary->safe_psql(
+	'postgres', q{
+	INSERT INTO mine SELECT generate_series(10,20) AS x;
+	SELECT pg_switch_wal();
+	CHECKPOINT;
+});
+
+$primary->poll_query_until('postgres',
+	q{ SELECT last_archived_wal FROM pg_stat_archiver },
+	$segment_name_2)
+  or die "Timed out while waiting for archiving to finish";
+
+# Test standby with archive_mode =3D on.
+my $standby1 =3D get_new_node('standby');
+$standby1->init_from_backup($primary, 'backup', has_restoring =3D> 1);
+$standby1->append_conf('postgresql.conf', "archive_mode =3D on");
+my $standby1_data =3D $standby1->data_dir;
+$standby1->start;
+$standby1->safe_psql('postgres', q{CHECKPOINT});
+
+# Recovery with archive_mode=3Don does not keep .ready signal files inheri=
ted
+# from backup.  Note that this WAL segment existed in the backup.
+ok( !-f "$standby1_data/$segment_path_1_ready",
+	".ready file for WAL segment $segment_name_1 present in backup got remove=
d with archive_mode=3Don on standby"
+);
+
+# Recovery with archive_mode=3Don should not create .ready files.
+# Note that this segment did not exist in the backup.
+ok( !-f "$standby1_data/$segment_path_2_ready",
+	".ready file for WAL segment $segment_name_2 not created on standby when =
archive_mode=3Don on standby"
+);
+
+# Recovery with archive_mode =3D on creates .done files.
+ok( -f "$standby1_data/$segment_path_2_done",
+	".done file for WAL segment $segment_name_2 created when archive_mode=3Do=
n on standby"
+);
+
+# Test recovery with archive_mode =3D always, which should always keep
+# .ready files if archiving is enabled, though here we want the archive
+# command to fail to persist the .ready files.  Note that this node
+# has inherited the archive command of the previous cold backup, that
+# will cause archiving failures.
+my $standby2 =3D get_new_node('standby2');
+$standby2->init_from_backup($primary, 'backup', has_restoring =3D> 1);
+$standby2->append_conf('postgresql.conf', 'archive_mode =3D always');
+my $standby2_data =3D $standby2->data_dir;
+$standby2->start;
+
+$standby2->safe_psql('postgres', q{CHECKPOINT});
+
+ok( -f "$standby2_data/$segment_path_1_ready",
+	".ready file for WAL segment $segment_name_1 existing in backup is kept w=
ith archive_mode=3Dalways on standby"
+);
+
+ok( -f "$standby2_data/$segment_path_2_ready",
+	".ready file for WAL segment $segment_name_2 created with archive_mode=3D=
always on standby"
+);
+
+# Reset statistics of the archiver for the next checks.
+$standby2->safe_psql('postgres', q{SELECT pg_stat_reset_shared('archiver')=
});
+
+# Now crash the cluster to check that recovery step does not
+# remove non-archived WAL segments on a standby where archiving
+# is enabled.
+$standby2->stop('immediate');
+$standby2->start;
+
+ok( -f "$standby2_data/$segment_path_1_ready",
+	"WAL segment still ready to archive after crash recovery on standby with =
archive_mode=3Dalways"
+);
+
+# Allow WAL archiving again, and wait for the segments to be archived.
+$standby2->safe_psql(
+	'postgres', q{
+	ALTER SYSTEM RESET archive_command;
+	SELECT pg_reload_conf();
+});
+$standby2->poll_query_until('postgres',
+	q{SELECT last_archived_wal FROM pg_stat_archiver},
+	$segment_name_2)
+  or die "Timed out while waiting for archiving to finish";
+
+is( $standby2->safe_psql(
+		'postgres', q{SELECT archived_count FROM pg_stat_archiver}),
+	'2',
+	'correct number of WAL segments archived from standby');
+
+ok( !-f "$standby2_data/$segment_path_1_ready"
+	  && !-f "$standby2_data/$segment_path_2_ready",
+	".ready files removed after archive success with archive_mode=3Dalways on=
 standby"
+);
+
+ok( -f "$standby2_data/$segment_path_1_done"
+	  && -f "$standby2_data/$segment_path_2_done",
+	".done files created after archive success with archive_mode=3Dalways on =
standby"
+);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs=
=2Elist
index 525d58e7f0..b319266bb3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1983,6 +1983,7 @@ RecordCacheEntry
 RecordCompareData
 RecordIOData
 RecoveryLockListsEntry
+RecoveryState
 RecoveryTargetTimeLineGoal
 RecoveryTargetType
 RectBox
--=20
2.26.0


--4Ckj6UjgE2iN1+kY--

--NDin8bjvE/0mNLFQ
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAl6dUJQACgkQnvQgOdby
QH2AVg//VUh2sNeIaF+mC7+l4aRhC7PtjWg6rXad5/uMxoa3QX3iDu0seL1CLfS1
Fvuy3TcqQcxFwMXnp+Rtp0XR8p3PnFohwlZxhA6/Wq1xJYO2kY4rD/seb8Mdi564
4qI5JTXpOB518ndmeth52pc70UxRFUnXgYre22PIZMV+7nF3K/LH5LA+q1TDzb/8
4INP0bJLUySFmuAmb6l7ha2lrS+FWwCbWegxl8xvak10LN2LvDqHZA0Wi3hT61gx
0T2ip3v3XAssPCoLYVC5+fWa1dw9C4O0p5+hsvtUZ9rjO18Q7iBBoB8eezc8i75D
biJmAT9Xq6j377+aeQJbNHvlK/jwejzr88J6GQoTIMQZ/D1yij1cE8TnNGtklQ2U
p8h2ZX3FedbMWMc/aHjUIs9JESNkWsnGJCE+oqr6nH4HdQsOarBWgGmfkDo6m4tE
wfBQrJKbFvxx2h8aaXBSvgNY1n4C0K9g43LvdTQaAtCWQONzdftZiJStmim+iVq3
8xw5GGpgMXe0+OOGscdNKvZJamTX3UUH2KDjteAVkYESF52AkowMNi4wRsI/uUG9
NvoRNCm98FJqTGBP4Gw36LUzTHb8Zs//X86qOhyI9+FyoLPyaRtuTSx1JAn3pVpe
fBgv/+gu/XFGHr/3I14g+rr5Jsb7O0JujWXZu4n317sZeMlu1KQ=
=7E5O
-----END PGP SIGNATURE-----

--NDin8bjvE/0mNLFQ--