Feedback-ID: i0fe9450f:Fastmail
Date: Tue, 21 Feb 2023 07:38:17 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>,
	Nathan Bossart <nathandbossart@gmail.com>,
	Thomas Munro <thomas.munro@gmail.com>,
	Fujii Masao <fujii@postgresql.org>,
	Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Weird failure with latches in curculio on v15[
Message-ID: <Y/P2WfDOMVw59/TY@paquier.xyz>
References: <20230209002413.GA603595@nathanxps13>
 <CA+TgmobE6rRs-tMArr733y9VucgNA5Nd9_qYgEVWjJDwHBwP-Q@mail.gmail.com>
 <3907196.1675957889@sss.pgh.pa.us>
 <CA+TgmoY6xxDdZ1YJjMb27Ei7qgFcbu=__WtDX_cOd3GQ5uvxWw@mail.gmail.com>
 <20230209192952.jvx56yuutlxuvjjf@awork3.anarazel.de>
 <CA+TgmobQ5iWVCoPokW1jBrRb3V_x0PUXD05vvgS3FqQecvXHMw@mail.gmail.com>
 <20230216163254.nktnbbb2b7wlpzwy@awork3.anarazel.de>
 <CA+TgmoamQWK3JYkYtf7GPuzh9en2bDPTzw9c3BF+cHZDt0y7AQ@mail.gmail.com>
 <20230218211522.grbzg75vzjtqafzf@awork3.anarazel.de>
 <CA+TgmoaLCxrdHPSnRLD1j1FQQx4k7QSJq-ybVOYj2aEF34sQLQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="V9Kha/1fwVVaO5is"
Content-Disposition: inline
In-Reply-To: <CA+TgmoaLCxrdHPSnRLD1j1FQQx4k7QSJq-ybVOYj2aEF34sQLQ@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/Y/P2WfDOMVw59/TY%40paquier.xyz>
Precedence: bulk


--V9Kha/1fwVVaO5is
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Sun, Feb 19, 2023 at 08:06:24PM +0530, Robert Haas wrote:
> I mean, my idea was to basically just have one big callback:
> ArchiverModuleMainLoopCB(). Which wouldn't return, or perhaps, would
> only return when archiving was totally caught up and there was nothing
> more to do right now. And then that callback could call functions like
> AreThereAnyMoreFilesIShouldBeArchivingAndIfYesWhatIsTheNextOne(). So
> it would call that function and it would find out about a file and
> start an HTTP session or whatever and then call that function again
> and start another HTTP session for the second file and so on until it
> had as much concurrency as it wanted. And then when it hit the
> concurrency limit, it would wait until at least one HTTP request
> finished. At that point it would call
> HeyEverybodyISuccessfullyArchivedAWalFile(), after which it could
> again ask for the next file and start a request for that one and so on
> and so forth.

This archiving implementation is not completely impossible with the
current API infrastructure, either?  If you consider the archiving as
a two-step process where segments are first copied into a cheap,
reliable area.  Then these could be pushed in block in a more remote
area like a S3 bucket?  Of course this depends on other things like
the cluster structure, but redundancy can be added with standby
archiving, as well.

I am not sure exactly how many requirements we want to push into a
callback, to be honest, and surely more requirements pushed to the
callback increases the odds of implementation mistakes, like a full
loop.  There already many ways to get it wrong with archiving, like
missing a flush of the archived segment before the callback returns to
ensure its durability..
--
Michael

--V9Kha/1fwVVaO5is
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmPz9lkACgkQnvQgOdby
QH233w//cyYNxefBZ7+VpiyUaH+AKZiCKpOGOvDo8K9RA8xDtFYkfgRKifK61J6s
/Zig5JiVJd0AME2Dr2aT9PWDNgdkGAvlsN9LS8QppRmTMR1Yc/AyWfjTzmrCR9+H
70sKnuNuS6a1iur1ZYFAo749G1gTe8+wKhHNId2MZYieXYHX9jaTHcSrBx+vsBw2
YLrpRlLGcTvcEf80d8XaYpAsy/K8zcSC8CSmYr5fSHgWrQYUk6l9xKY7UTPOGNGE
T4kusZAeLQlZGp1RiFMaFpQQFBb2moQ1F3LTeq2ywi7Z25yIVBhVxBR3X4DJpt3b
qXevPck6fczIHB75SOQKA6F3wnj9gFMUwPBJ5sfUsEoUQ3Zv8D1ykhJYh1P7RNwz
x6JP9VdKkpYeI5JA6KX2vpTFA9iR5EodVdafYJUMn3y8TPU7XqjBzQ03SaIYxMAU
tncybB1WFr1FTFB8X/lWyGat6ZYfQUVBQVfiyqKIJ5U2+pGNn5oVoF6cRgE6mhj1
gPK7/q93Ivv+sMy/bYmocf4eF7P5H9v5d2UqVc5JzI++LW4XL9ABSzIFXESj2aVH
n3lGe9gtrtwkeT9JcHqgfQ4IZyAzTlUMknY4gC3qhOww0dI6sHkwCesS7BfAc4gc
HKts+Alki0gjdGGXLymDzxdD5Pq2V6olacAQyHT6Kee/cCA9xBE=
=M8Ee
-----END PGP SIGNATURE-----

--V9Kha/1fwVVaO5is--