Feedback-ID: i0fe9450f:Fastmail
Date: Wed, 8 Jan 2025 08:47:05 +0900
From: Michael Paquier <michael@paquier.xyz>
To: "Andrey M. Borodin" <x4mmm@yandex-team.ru>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>,
	Nathan Bossart <nathandbossart@gmail.com>,
	Japin Li <japinli@hotmail.com>,
	Ian Lawrence Barwick <barwick@gmail.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	Cary Huang <cary.huang@highgo.ca>,
	SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: Switching XLog source from archive to streaming when primary
 available
Message-ID: <Z328-dr7yQLdNdMk@paquier.xyz>
References: 
 <CALj2ACXV9=q5mXbPTgk9gKPQroxjpgyP7VKQnA1FREQO0iRXiA@mail.gmail.com>
 <20240305020452.GA3373526@nathanxps13>
 <CALj2ACVYVgq+QxeLmhY5DO+D6+7PwerdsVTeBxPuSeFyrgOPtg@mail.gmail.com>
 <20240305195212.GB3481820@nathanxps13>
 <CALj2ACVict-k-r94VYkJ3RB8=b-=hotS7N=e2_PmndFEke2r8w@mail.gmail.com>
 <20240306161926.GB3542434@nathanxps13>
 <CALj2ACU32_N+ymdW01WHTP+xEr8yVi9u1sFoqqg3kL4u36Mx5A@mail.gmail.com>
 <ZffaQt7UbM2Q9kYh@paquier.xyz>
 <CALj2ACVVM8c9pbCgmyxvViFy=f==QNHhPFSGjvq0Sz7Y0beZsw@mail.gmail.com>
 <2D6CD551-8110-450B-BBD9-DAE64822304D@yandex-team.ru>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="F0Z65172267kNtfu"
Content-Disposition: inline
In-Reply-To: <2D6CD551-8110-450B-BBD9-DAE64822304D@yandex-team.ru>
Archived-At: 
 <https://www.postgresql.org/message-id/Z328-dr7yQLdNdMk%40paquier.xyz>
Precedence: bulk


--F0Z65172267kNtfu
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Jan 02, 2025 at 11:12:37PM +0500, Andrey M. Borodin wrote:
> In my observation restore from archive is many orders of magnitude
> faster than streaming replication. Advanced archive tools employ
> compression (x6 to speed), download parallelism (x4), are not
> constrained be primary's network limits (x3) and disk limits, do not
> depend on complicated FEBE protocol, etc.

This is a fair argument in terms of flexibility of what can be
achieved on a file-basis, yes, because you are not bottlenecked by
the existing replication protocol and can request them ahead of time
if necessary and can decide what you want within a single
restore_command or archive_command (or module for the latter).

It may be relevant to think in terms of what could be done at protocol
level to retrieve batches of WAL segments so as the backend has a
better control on how each segment is handled in a batch, or provide
better in-core tools to achieve that with the existing two command
GUCs for restore and archiving?  Nathan has also proposed a couple of
months ago restore modules, because relying on commands can be very
fancy in terms of error handling.  And we already have the archive
module part.
--
Michael

--F0Z65172267kNtfu
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmd9vPkACgkQnvQgOdby
QH1TNg//egcoX5cAathGLRRn3WT3Ucv8QcYRHruerJLzIyzlJZWWRPL9ymh1dLsK
m9zuiVUvCehtxHgJMyCNY+xLtz1fL4nFmv2Pxk+yd4MqVTOssHQf/qGioKlaMVEq
9GKVJyEVKhl/cYNu9CoWSoUQ6XFfE7+iTJutTUBFAP0PXVScLerrAZIEM7dQBJ5R
3S/g38h7p4RxRF8DCKHraIVZpBlPLiOs/MG7kTZZxLgfSprP9RcKcxeYu8rEMJwt
rRqNft0uJznt2oZVgiiN8Di+nTQ+56w6USQEx1+qfhSqRICxWPV+CS7Pt0e2Bkx/
oUZO8YY12/lV5kPZSeJESfrSPPKt88oV/kwZ3oLMPfWaEJWFKQus6dSrebTeqn7x
8hE2RRfy5IYsT9M5hjT30pepPBsvHXKVYQKQOno65qbwM4/NdtclO9eprOBWsij4
TjusUqTCUc3+GyRl9Y6ZJt9bxBWfkWidhvqRDqVF9I55DWpYazYKxxYeC4X5DriE
whZ/dauCgzUymTWXsYLP9dJOUE87UYqLTAhU5UlVT4mqkNbkv77D2OXOEZCBfNdC
bkyth+cNAsokIPqkTc1OjEhxme+t8aAoEDXsh8X51+wGkvmWIZKLw5x9iCueHRxu
8v3stcqp8DRl2EyznyYHZ7YHi+XONXOHRp0OPIsgCN1lqHXjH0k=
=RLxG
-----END PGP SIGNATURE-----

--F0Z65172267kNtfu--