public inbox for [email protected]
help / color / mirror / Atom feedFrom: Michael Paquier <[email protected]>
To: Robert Haas <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Nathan Bossart <[email protected]>
Cc: Thomas Munro <[email protected]>
Cc: Fujii Masao <[email protected]>
Cc: Postgres hackers <[email protected]>
Subject: Re: Weird failure with latches in curculio on v15[
Date: Tue, 21 Feb 2023 07:38:17 +0900
Message-ID: <Y/P2WfDOMVw59/[email protected]> (raw)
In-Reply-To: <CA+TgmoaLCxrdHPSnRLD1j1FQQx4k7QSJq-ybVOYj2aEF34sQLQ@mail.gmail.com>
References: <20230209002413.GA603595@nathanxps13>
<CA+TgmobE6rRs-tMArr733y9VucgNA5Nd9_qYgEVWjJDwHBwP-Q@mail.gmail.com>
<[email protected]>
<CA+TgmoY6xxDdZ1YJjMb27Ei7qgFcbu=__WtDX_cOd3GQ5uvxWw@mail.gmail.com>
<[email protected]>
<CA+TgmobQ5iWVCoPokW1jBrRb3V_x0PUXD05vvgS3FqQecvXHMw@mail.gmail.com>
<[email protected]>
<CA+TgmoamQWK3JYkYtf7GPuzh9en2bDPTzw9c3BF+cHZDt0y7AQ@mail.gmail.com>
<[email protected]>
<CA+TgmoaLCxrdHPSnRLD1j1FQQx4k7QSJq-ybVOYj2aEF34sQLQ@mail.gmail.com>
On Sun, Feb 19, 2023 at 08:06:24PM +0530, Robert Haas wrote:
> I mean, my idea was to basically just have one big callback:
> ArchiverModuleMainLoopCB(). Which wouldn't return, or perhaps, would
> only return when archiving was totally caught up and there was nothing
> more to do right now. And then that callback could call functions like
> AreThereAnyMoreFilesIShouldBeArchivingAndIfYesWhatIsTheNextOne(). So
> it would call that function and it would find out about a file and
> start an HTTP session or whatever and then call that function again
> and start another HTTP session for the second file and so on until it
> had as much concurrency as it wanted. And then when it hit the
> concurrency limit, it would wait until at least one HTTP request
> finished. At that point it would call
> HeyEverybodyISuccessfullyArchivedAWalFile(), after which it could
> again ask for the next file and start a request for that one and so on
> and so forth.
This archiving implementation is not completely impossible with the
current API infrastructure, either? If you consider the archiving as
a two-step process where segments are first copied into a cheap,
reliable area. Then these could be pushed in block in a more remote
area like a S3 bucket? Of course this depends on other things like
the cluster structure, but redundancy can be added with standby
archiving, as well.
I am not sure exactly how many requirements we want to push into a
callback, to be honest, and surely more requirements pushed to the
callback increases the odds of implementation mistakes, like a full
loop. There already many ways to get it wrong with archiving, like
missing a flush of the archived segment before the callback returns to
ensure its durability..
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../Y%2FP2WfDOMVw59%[email protected]/2-signature.asc)
download
view thread (25+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Weird failure with latches in curculio on v15[
In-Reply-To: <Y/P2WfDOMVw59/[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox