From: Scott Whitney <scott@journyx.com>
To: pgsql-admin@postgresql.org
Subject: Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times
Date: Fri, 1 May 2015 10:29:33 -0500 (CDT)
Message-ID: <1652559419.2123167.1430494173076.JavaMail.zimbra@mail.int.journyx.com> (raw)
In-Reply-To: <CF30A25D-6A2D-46B7-831F-D32C1478C882@elevated-dev.com>
References: <em5e30e63a-661c-44ad-8996-601c945efc8b@oyster-creek>
<CF30A25D-6A2D-46B7-831F-D32C1478C882@elevated-dev.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-admin>
A few years back, I wrote something very similar to what ended up being in PG, and I've been using it successfully ever since.
I did find there's a law of diminishing returns after about 6 threads running pg_dump.
In my case, I'm on a 6-drive SSD RAID and getting nowhere near the IOPS of the RAID,
but my suspicion is either that I'm maxing my RAID controller's potential or pg_dump just
won't get any faster.
16 cores 3.4GHz in my case.
Now, the interesting thing is that I tried this on spindle arrays, single drives, multiple machines (quad core,
quad proc, dual core, dual core-dual proc, etc) and there always seemed to be a diminishing return around
6 or 8 jobs running.
----- On May 1, 2015, at 10:22 AM, Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On May 1, 2015, at 9:16 AM, Susan K. McClure <smcclure@rice.edu> wrote:
> > The DB in question is ~25GB. The processor has 24 Cpus, 12 cores
>> I have tried with "--jobs = 8, 12, and 20" with little or no discernible
> > improvements.
> I would immediately suspect that you might be disk-bound instead of CPU-bound.
> What’s the setup of your storage system?
> --
> Scott Ribe
> scott_ribe@elevated-dev.com
> http://www.elevated-dev.com/
> https://www.linkedin.com/in/scottribe/
> (303) 722-0567 voice
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
Journyx, Inc.
7600 Burnet Road #300
Austin, TX 78757
www.journyx.com
p 512.834.8888
f 512.834.8858
Do you receive our promotional emails? You can subscribe or unsubscribe to those emails at http://go.journyx.com/emailPreference/e/4932/714
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: scott@journyx.com
Subject: Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times
In-Reply-To: <1652559419.2123167.1430494173076.JavaMail.zimbra@mail.int.journyx.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox