Feedback-ID: id4a34324:Fastmail
Date: Sun, 5 Feb 2023 16:07:50 -0800
From: Andres Freund <andres@anarazel.de>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>, Tom Lane <tgl@sss.pgh.pa.us>,
	Thomas Munro <thomas.munro@gmail.com>,
	Fujii Masao <fujii@postgresql.org>,
	Postgres hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Weird failure with latches in curculio on v15
Message-ID: <20230206000750.ohqjmbfo3trtwvhc@alap3.anarazel.de>
References: <20230201165801.33ydbxvjdbomjqa7@alap3.anarazel.de>
 <20230201175806.GA3199959@nathanxps13>
 <20230201223555.GA3721373@nathanxps13>
 <Y9sam108o4mxZFiS@paquier.xyz>
 <20230204113029.xlcqrbxhp6lerrnc@alap3.anarazel.de>
 <20230204180354.GA258107@nathanxps13>
 <Y979NR3U5VnWrTwB@paquier.xyz>
 <20230205221938.GA274245@nathanxps13>
 <20230205230157.45kl5gqryupzdoyb@alap3.anarazel.de>
 <20230205235747.GA275913@nathanxps13>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230205235747.GA275913@nathanxps13>
Archived-At: <https://www.postgresql.org/message-id/20230206000750.ohqjmbfo3trtwvhc%40alap3.anarazel.de>
Precedence: bulk

Hi,

On 2023-02-05 15:57:47 -0800, Nathan Bossart wrote:
> > For the segment files, we'd likely need a parameter to indicate whether
> > the restore is random or not.
> 
> Wouldn't this approach still require each module to handle restoring ahead
> of time?

Yes, to some degree at least. I was just describing a few pretty obvious
improvements.

The core code can make that a lot easier though. The problem of where to
store such files can be provided by core code (presumably a separate
directory). A GUC for aggressiveness can be provided. Etc.


> I agree that the shell overhead isn't the main performance issue,
> but it's unclear to me how much of this should be baked into
> PostgreSQL.

I don't know fully either. But just reimplementing all of it in
different modules doesn't seem like a sane approach either. A lot of it
is policy that we need to solve once, centrally.


> I mean, we could introduce a GUC that tells us how far ahead to
> restore and have a background worker (or multiple background workers)
> asynchronously pull files into a staging directory via the callbacks.
> Is that the sort of scope you are envisioning?

Closer, at least.

Greetings,

Andres Freund