Date: Thu, 9 Feb 2023 09:23:08 -0800
From: Nathan Bossart <nathandbossart@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Michael Paquier <michael@paquier.xyz>,
	Andres Freund <andres@anarazel.de>,
	Thomas Munro <thomas.munro@gmail.com>,
	Fujii Masao <fujii@postgresql.org>,
	Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Weird failure with latches in curculio on v15
Message-ID: <20230209172308.GA605342@nathanxps13>
References: <20230206004631.GA277434@nathanxps13>
 <CA+TgmoYMk+NtLxU8xpAvWe_6K783d2Pr09cUpUJFRChdj_rcbQ@mail.gmail.com>
 <20230208174350.GB451849@nathanxps13>
 <CA+TgmobY5ALQE4U2q32YohLbJfze8CDBZ3rs2x2+ccAa+Pqw6g@mail.gmail.com>
 <20230208222554.GB546776@nathanxps13>
 <Y+Q2qCecB8zcqF9S@paquier.xyz>
 <20230209002413.GA603595@nathanxps13>
 <CA+TgmobE6rRs-tMArr733y9VucgNA5Nd9_qYgEVWjJDwHBwP-Q@mail.gmail.com>
 <3907196.1675957889@sss.pgh.pa.us>
 <CA+TgmoY6xxDdZ1YJjMb27Ei7qgFcbu=__WtDX_cOd3GQ5uvxWw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CA+TgmoY6xxDdZ1YJjMb27Ei7qgFcbu=__WtDX_cOd3GQ5uvxWw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/20230209172308.GA605342%40nathanxps13>
Precedence: bulk

On Thu, Feb 09, 2023 at 11:12:21AM -0500, Robert Haas wrote:
> On Thu, Feb 9, 2023 at 10:51 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> If we think we need primitives to let the archiver hooks get all
>> the pending files, or whatever, by all means add those.  But don't
>> cede fundamental control of the archiver.  The hooks need to be
>> decoration on a framework we provide, not the framework themselves.
> 
> I don't quite see how you can make asynchronous and parallel archiving
> work if the archiver process only calls into the archive module at
> times that it chooses. That would mean that the module has to return
> control to the archiver when it's in the middle of archiving one or
> more files -- and then I don't see how it can get control back at the
> appropriate time. Do you have a thought about that?

I've been thinking about this, actually.  I'm wondering if we could provide
a list of files to the archiving callback (configurable via a variable in
ArchiveModuleState), and then have the callback return a list of files that
are archived.  (Or maybe we just put the list of files that need archiving
in ArchiveModuleState.)  The returned list could include files that were
sent to the callback previously.  The archive module would be responsible
for creating background worker(s) (if desired), dispatching files
to-be-archived to its background worker(s), and gathering the list of
archived files to return.

This is admittedly half-formed, but I'm tempted to hack something together
quickly to see whether it might be viable.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com