public inbox for [email protected]
help / color / mirror / Atom feedFrom: Jacob Champion <[email protected]>
To: [email protected] <[email protected]>
To: [email protected] <[email protected]>
Subject: Re: pg_upgrade parallelism
Date: Wed, 17 Nov 2021 20:04:41 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <YZVbtHKYP02AZDIO@ahch-to>
References: <YZVbtHKYP02AZDIO@ahch-to>
On Wed, 2021-11-17 at 14:44 -0500, Jaime Casanova wrote:
> I'm trying to add more parallelism by copying individual segments
> of a relfilenode in different processes. Does anyone one see a big
> problem in trying to do that? I'm asking because no one did it before,
> that could not be a good sign.
I looked into speeding this up a while back, too. For the use case I
was looking at -- Greenplum, which has huge numbers of relfilenodes --
spinning disk I/O was absolutely the bottleneck and that is typically
not easily parallelizable. (In fact I felt at the time that Andres'
work on async I/O might be a better way forward, at least for some
filesystems.)
But you mentioned that you were seeing disks that weren't saturated, so
maybe some CPU optimization is still valuable? I am a little skeptical
that more parallelism is the way to do that, but numbers trump my
skepticism.
> - why we read()/write() at all? is not a faster way of copying the file?
> i'm asking that because i don't actually know.
I have idly wondered if something based on splice() would be faster,
but I haven't actually tried it.
But there is now support for copy-on-write with the clone mode, isn't
there? Or are you not able to take advantage of it?
--Jacob
view thread (8+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: pg_upgrade parallelism
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox