Date: Wed, 5 May 2021 16:13:08 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Message-ID: <20210505201308.GI20766@tamriel.snowman.net>
References: 
 <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
 <20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
 <CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
 <CAFiTN-tR+3+GjP0Qeys8jwh=jz9VpP2ibhT9ubFDLmgNb1QtMg@mail.gmail.com>
 <CA+TgmoZHBb-TmwYtAzoPYu2FW_NWyLOnABNz_U-=59wTJuz4ig@mail.gmail.com>
 <20210505170601.GF20766@tamriel.snowman.net>
 <CA+Tgmoa4=k3CeGh+OX0D9+p7Nmj3ZYaeD2ahk+=n0Bz7m5d_HA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="jZItKTd6fsMyMO5a"
Content-Disposition: inline
In-Reply-To: 
 <CA+Tgmoa4=k3CeGh+OX0D9+p7Nmj3ZYaeD2ahk+=n0Bz7m5d_HA@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20210505201308.GI20766%40tamriel.snowman.net>
Precedence: bulk


--jZItKTd6fsMyMO5a
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Wed, May 5, 2021 at 1:06 PM Stephen Frost <sfrost@snowman.net> wrote:
> > It's not just about making sure that we archive the history file for a
> > timeline before archiving WAL segments along that timeline but also
> > about making sure we get that history file into the archive as fast as
> > we can, and archiving a 16MB WAL first would certainly delay that.
>=20
> Ooph. That's a rather tough constraint. Could we get around it by
> introducing some kind of signalling mechanism, perhaps? Like if
> there's a new history file, that must mean the server has switched
> timelines -- I think, anyway -- so if we notified the archiver every
> time there was a timeline switch it could react accordingly.

I would think something like that would be alright and not worse than
what we've got now.

That said, in an ideal world, we'd have a way to get the new timeline to
switch to in a way that doesn't leave open race conditions, so as long
we're talking about big changes to the way archiving and archive_command
work (or about throwing out the horrible idea that is archive_command in
the first place and replacing it with appropriate hooks such that
someone could install an extension which would handle archiving...), I
would hope we'd have a way of saying "please, atomically, go get me a new
timeline."

Just as a reminder for those following along at home, as I'm sure you're
already aware, the way we figure out what timeline to switch to when a
replica is getting promoted is that we go run the restore command asking
for history files until we get back "nope, there is no file named
0000123.history", and then we switch to that timeline and then try to
push such a history file into the repo and hope that it works.

Thanks,

Stephen

--jZItKTd6fsMyMO5a
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJgkvxUAAoJEO1sijiDR2RVFtEP/3/Cvr+Hff9MRFkAOvVhcc2k
PcxX+sieAmxhsGKP8yoH+jirBKTDtUhUreCG0pGyL/qQpAlqEy5SqvOkuh9B9E2h
faZgFLPjDGaO3BU1VYQ8YVOHLVFV3voPoIoaCVLp/g/t4zyJxxnjNtGiRuy/Qx8b
OfQRdOOB6obUElpHy5JRlNPJZNnxWmzm/9NezLMJb/wWxD97EGH/WoyKgCO8IOUg
7yW6TI5gYq7xBPuB6JqBreTB1ODE+TlbGK6CnhywjEPRuNFUuFEQOxmHe1UNZh7E
q3K/5SX75M8k/UwlNqLZ5/ozjydowa9TCalRWeiuWgtYp8eaDDUIo0u3OTVgMhhj
SFAIZrKadgHJMcUSLlxZKjOHGNYUTe97kJT12wEDCTC0K+9RBYMPdudnPtB+GmQt
eGQix0XC1ZLw3hAFjc/QL9weXraX1iCTZzL81OLkoIEVstIY6R3PMsHrTY0PQ8Bz
bMp/qumjnB65bv7Jhmk22PMXAO13MtFZ5vKxcg+wbZiwjKcMyiQMOnZ1n2ABOjq6
61VOtIvimrnC87pl7cGKYT9lPbcoA0t6OUVZ69KBJgdEWTEkoPR3u42/XFn00sz/
BY/W4RnPW7SdqOtuwCUNeV11CnKeLF2W//oLiYr64JWFgsLBpMPYfS6/lSzdzIo3
bNqjBAjvYXtGwAcLWgjd
=QH8Q
-----END PGP SIGNATURE-----

--jZItKTd6fsMyMO5a--