public inbox for [email protected]  

help / color / mirror / Atom feed

From: Nathan Bossart <[email protected]>
To: Robert Haas <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Thomas Munro <[email protected]>
Cc: Fujii Masao <[email protected]>
Cc: Postgres hackers <[email protected]>
Subject: Re: Weird failure with latches in curculio on v15
Date: Wed, 8 Feb 2023 09:43:50 -0800
Message-ID: <20230208174350.GB451849@nathanxps13> (raw)
In-Reply-To: <CA+TgmoYMk+NtLxU8xpAvWe_6K783d2Pr09cUpUJFRChdj_rcbQ@mail.gmail.com>
References: <[email protected]>
	<[email protected]>
	<20230204180354.GA258107@nathanxps13>
	<[email protected]>
	<20230205221938.GA274245@nathanxps13>
	<[email protected]>
	<20230205235747.GA275913@nathanxps13>
	<[email protected]>
	<20230206004631.GA277434@nathanxps13>
	<CA+TgmoYMk+NtLxU8xpAvWe_6K783d2Pr09cUpUJFRChdj_rcbQ@mail.gmail.com>

On Wed, Feb 08, 2023 at 10:22:24AM -0500, Robert Haas wrote:
> I felt like the archive modules work was a step forward when we did,
> because basic_archive does some things that you're not likely to get
> right if you do it on your own. And a similar approach to
> restore_command might be also be valuable, at least in my opinion.
> However, the gains that we can get out of the archive module facility
> in its present form do seem to be somewhat limited, for exactly the
> kinds of reasons being discussed here.

I'm glad to hear that there is interest in taking this stuff to the next
level.  I'm currently planning to first get the basic API in place for
recovery modules like we have for archive modules, but I'm hoping to
position it so that it leads naturally to asynchronous, parallel, and/or
batching approaches down the road (v17?).

> I kind of wonder whether we ought to try to flip the model around. At
> present, the idea is that the archiver is doing its thing and it makes
> callbacks into the archive module. But what if we got rid of the
> archiver main loop altogether and put the main loop inside of the
> archive module, and have it call back to some functions that we
> provide? One function could be like char
> *pgarch_next_file_to_be_archived_if_there_is_one_ready(void) and the
> other could be like void
> pgarch_some_file_that_you_gave_me_previously_is_now_fully_archived(char
> *which_one). That way, we'd break the tight coupling where you have to
> get a unit of work and perform it in full before you can get the next
> unit of work. Some variant of this could work on the restore side,
> too, I think, although we have less certainty about how much it makes
> to prefetch for restore than we do about what needs to be archived.

I think this could be a good approach if we decide not to bake too much
into PostgreSQL itself (e.g., such as creating multiple archive workers
that each call out to the module).  Archive module authors would
effectively need to write their own archiver processes.  That sounds super
flexible, but it also sounds like it might be harder to get right.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

view thread (15+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Weird failure with latches in curculio on v15
  In-Reply-To: <20230208174350.GB451849@nathanxps13>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox