Date: Wed, 5 May 2021 16:32:09 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Message-ID: <20210505203209.GJ20766@tamriel.snowman.net>
References: 
 <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
 <20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
 <CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
 <CAFiTN-tR+3+GjP0Qeys8jwh=jz9VpP2ibhT9ubFDLmgNb1QtMg@mail.gmail.com>
 <CA+TgmoZHBb-TmwYtAzoPYu2FW_NWyLOnABNz_U-=59wTJuz4ig@mail.gmail.com>
 <20210505170601.GF20766@tamriel.snowman.net>
 <CA+Tgmoa4=k3CeGh+OX0D9+p7Nmj3ZYaeD2ahk+=n0Bz7m5d_HA@mail.gmail.com>
 <20210505201308.GI20766@tamriel.snowman.net>
 <CA+TgmoY5hS-CXQYumdoXyY0fyxXS+Sgi6odVdvu8Ri=5LkyU_g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="nKGX6YqGv7roYD47"
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmoY5hS-CXQYumdoXyY0fyxXS+Sgi6odVdvu8Ri=5LkyU_g@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20210505203209.GJ20766%40tamriel.snowman.net>
Precedence: bulk


--nKGX6YqGv7roYD47
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Wed, May 5, 2021 at 4:13 PM Stephen Frost <sfrost@snowman.net> wrote:
> > That said, in an ideal world, we'd have a way to get the new timeline to
> > switch to in a way that doesn't leave open race conditions, so as long
> > we're talking about big changes to the way archiving and archive_command
> > work (or about throwing out the horrible idea that is archive_command in
> > the first place and replacing it with appropriate hooks such that
> > someone could install an extension which would handle archiving...), I
> > would hope we'd have a way of saying "please, atomically, go get me a n=
ew
> > timeline."
> >
> > Just as a reminder for those following along at home, as I'm sure you're
> > already aware, the way we figure out what timeline to switch to when a
> > replica is getting promoted is that we go run the restore command asking
> > for history files until we get back "nope, there is no file named
> > 0000123.history", and then we switch to that timeline and then try to
> > push such a history file into the repo and hope that it works.
>=20
> Huh, I had not thought about that problem. So, at the risk of getting
> sidetracked, what exactly are you asking for here? Let the extension
> pick the timeline using an algorithm of its own devising, rather than
> having core do it? Or what?

Having the extension do it somehow is an interesting idea and one which
might be kind of cool.

The first thought I had was to make it archive_command's job to "pick"
the timeline by just re-trying to push the .history file (the actual
contents of it don't change, as the information in the file is about the
timeline we are switching *from* and at what LSN).  That requires an
archive command which will fail if that file already exists though and,
ideally, would perform the file archival in an atomic fashion (though
this last bit isn't stricly necessary- anything along these lines would
certainly be better than the current state).

Having an entirely independent command/hook that's explicitly for this
case would be another approach, of course, either in a manner that
allows the extension to pick the destination timeline or is defined to
be "return success only if the file is successfully archived, but do
*not* overwrite any existing file of the same name and return an error
instead." and then the same approach as outlined above.

Thanks,

Stephen

--nKGX6YqGv7roYD47
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJgkwDIAAoJEO1sijiDR2RVmY0QAKZvqCTe/Mlcd9OKTjR0LZZN
/zjVCADOkWaV0iXSt1hc5FXEyQts0trIVzN/hS4vXkpspUbV01JoA0eSdCwuaHII
ADVYxJdVe699fCuijHfOe1UyAzgZAYdJP9u/mr4d+eKjzTmhP52sk1pcO643OdXx
uvHgzHLP7o5lOeQRwWbxq5MmSaYg6xW4j1oi6jAjR3zSYgyBDqCKalxnBhASb5SG
vsqBAlAdTjfWWkUHtccer0i3vfJztmLEmf4HHVl8hpU4fo7z5u/ZeciuzcnkyPJs
xbzc10s7ByiCtjuJXdbbeSspJrJ3dwvh/97z/QIEYE+RYizb9V8Br3yzBm7b1EDV
H7lGwJ8klaf/KgoDTn77WgErEb951vRwb4aUM2QSlCLQVp3gKi6eceLhmEnehJrh
pmfUYoD6H505O9vdsJHigoh1/Wx6OLe331nbJm56uV3+3pEBZyQo8dKiVCpxzR2U
GAXcJPrqqAA2F2OvVp2iCqq6sM6fNLshWxBOsiWPjO6CVx4mJDxg0Gq8W16TzRg9
7w8FfEEo59CnZntYJBCe+/I6s8MR7rik49BAzk+88fe1rXRYijl1h3gQ25+TGsex
Zcmd94VpNf3f9sJiWGBK6ebDUY4ezdFihkC6vAAXixQpxJ9R3VTrEtjFqukK8djq
TNnx846vY34bqiXPbklv
=pZZS
-----END PGP SIGNATURE-----

--nKGX6YqGv7roYD47--