Date: Wed, 8 Feb 2023 16:24:13 -0800
From: Nathan Bossart <nathandbossart@gmail.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Robert Haas <robertmhaas@gmail.com>, Andres Freund <andres@anarazel.de>,
	Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@gmail.com>,
	Fujii Masao <fujii@postgresql.org>,
	Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Weird failure with latches in curculio on v15
Message-ID: <20230209002413.GA603595@nathanxps13>
References: <20230205221938.GA274245@nathanxps13>
 <20230205230157.45kl5gqryupzdoyb@alap3.anarazel.de>
 <20230205235747.GA275913@nathanxps13>
 <20230206000750.ohqjmbfo3trtwvhc@alap3.anarazel.de>
 <20230206004631.GA277434@nathanxps13>
 <CA+TgmoYMk+NtLxU8xpAvWe_6K783d2Pr09cUpUJFRChdj_rcbQ@mail.gmail.com>
 <20230208174350.GB451849@nathanxps13>
 <CA+TgmobY5ALQE4U2q32YohLbJfze8CDBZ3rs2x2+ccAa+Pqw6g@mail.gmail.com>
 <20230208222554.GB546776@nathanxps13>
 <Y+Q2qCecB8zcqF9S@paquier.xyz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Y+Q2qCecB8zcqF9S@paquier.xyz>
Archived-At: <https://www.postgresql.org/message-id/20230209002413.GA603595%40nathanxps13>
Precedence: bulk

On Thu, Feb 09, 2023 at 08:56:24AM +0900, Michael Paquier wrote:
> On Wed, Feb 08, 2023 at 02:25:54PM -0800, Nathan Bossart wrote:
>> These are all good points.  Perhaps there could be a base archiver
>> implementation that shell_archive uses (and that other modules could use if
>> desired, which might be important for backward compatibility with the
>> existing callbacks).  But if you want to do something fancier than
>> archiving sequentially, you could write your own.
> 
> Which is basically the kind of things you can already achieve with a
> background worker and a module of your own?

IMO one of the big pieces that's missing is a way to get the next N files
to archive.  Right now, you'd have to trawl through archive_status on your
own if you wanted to batch/parallelize.  I think one advantage of what
Robert is suggesting is that we could easily provide a supported way to get
the next set of files to archive, and we can asynchronously mark them
"done".  Otherwise, each module has to implement this.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com