Date: Wed, 5 May 2021 13:06:01 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Message-ID: <20210505170601.GF20766@tamriel.snowman.net>
References: 
 <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
 <20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
 <CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
 <CAFiTN-tR+3+GjP0Qeys8jwh=jz9VpP2ibhT9ubFDLmgNb1QtMg@mail.gmail.com>
 <CA+TgmoZHBb-TmwYtAzoPYu2FW_NWyLOnABNz_U-=59wTJuz4ig@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="MiIoAWmx/Qw6/Nsd"
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmoZHBb-TmwYtAzoPYu2FW_NWyLOnABNz_U-=59wTJuz4ig@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20210505170601.GF20766%40tamriel.snowman.net>
Precedence: bulk


--MiIoAWmx/Qw6/Nsd
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Tue, May 4, 2021 at 11:54 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > I agree that if we continue to archive one file using the archive
> > command then Robert's solution of checking the existence of the next
> > WAL segment (N+1) has an advantage.  But, currently, if you notice
> > pgarch_readyXlog always consider any history file as the oldest file
> > but that will not be true if we try to predict the next WAL segment
> > name.  For example, if we have archived 000000010000000000000004 then
> > next we will look for 000000010000000000000005 but after generating
> > segment 000000010000000000000005, if there is a timeline switch then
> > we will have the below files in the archive status
> > (000000010000000000000005.ready, 00000002.history file).  Now, the
> > existing archiver will archive 00000002.history first whereas our code
> > will archive 000000010000000000000005 first.  Said that I don't see
> > any problem with that because before archiving any segment file from
> > TL 2 we will definitely archive the 00000002.history file because we
> > will not find the 000000010000000000000006.ready and we will scan the
> > full directory and now we will find 00000002.history as oldest file.
>=20
> OK, that makes sense and is good to know.

I expect David will chime in on this thread too, but I did want to point
out that when it coming to archiving history files you'd *really* like
that to be done just about as quickly as absolutely possible, to avoid
the case that we saw before that code was added, to wit: two promotions
done too quickly that ended up with conflicting history and possibly
conflicting WAL files trying to be archived, and ensuing madness.

It's not just about making sure that we archive the history file for a
timeline before archiving WAL segments along that timeline but also
about making sure we get that history file into the archive as fast as
we can, and archiving a 16MB WAL first would certainly delay that.

Thanks,

Stephen

--MiIoAWmx/Qw6/Nsd
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJgktB4AAoJEO1sijiDR2RV5BsP/2V6kKnJHy5l+qmtN2bp2Wv/
g9rB5xgs/PhCOy9IeIFo+UHoL1cMxeqUTVQJ1ouI2INfdkzpjnuTYQbrpNrmtT5P
9APdpD+lfRKKrzXvHx/Mj0mPmI9AhnSiZ1yAiuBkE0PDSvlDqn5KEQM6lGUs94Vr
qjWvgctch198UkcP4yF180FCyoZ6IlpwjB48UTbiDL+kxh5mi2EAMmrZg+ol+4bW
SaHU8/T7dnkbvIFLCIYx2RECkrwB8hPqigyIDCLwCE4avD23nW/jpGhRtZRlVXI2
huNiVl4bECXib+eruuaFoxPGY2IbC97emuL1151BkxPGAejz9rh2U1pRoaSuGDOX
HLf07GU1Q8WcHOXYwkAUgSiqGXzvpIc7Db3NXTdEzXQQELabG+OQfBaEFybgOzlC
BneFho4+GnfS3mEqCWUHtoFF1rRp8IWEKkevsGhLIxxCvUw8kBSMXIwCzPfojFHF
iVEpc2nI1leijekPy6oPpJFnI6K1SRHIAAlteHFi6/zf6a/AY/qisw7+shLg01fz
iooU8aMVNeta76GadjQr7kFduimOkTFMHqjb5+p+EanRcMSB0nk/tplSFRJjs8qy
R7/joYfuWF/CzSUsE1hOckPFTWB2eYnm7SU8YcmZWMW0N9HbwEUrvcYovE1UU0r5
QykZ/MQscSEir2AOKTMr
=O7BF
-----END PGP SIGNATURE-----

--MiIoAWmx/Qw6/Nsd--