Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1YoCsv-0000GJ-Ft for pgsql-admin@arkaria.postgresql.org; Fri, 01 May 2015 15:29:49 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1YoCsu-0006ZL-8T for pgsql-admin@arkaria.postgresql.org; Fri, 01 May 2015 15:29:48 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1YoCsn-0006K6-3u for pgsql-admin@postgresql.org; Fri, 01 May 2015 15:29:41 +0000 Received: from mail.journyx.com ([209.163.206.41] helo=mail.int.journyx.com) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1YoCsh-0003Re-HW for pgsql-admin@postgresql.org; Fri, 01 May 2015 15:29:37 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.int.journyx.com (Postfix) with ESMTP id B7708300357 for ; Fri, 1 May 2015 10:29:33 -0500 (CDT) Received: from mail.int.journyx.com ([127.0.0.1]) by localhost (mail.int.journyx.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id DmsQT_1uvlww for ; Fri, 1 May 2015 10:29:33 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.int.journyx.com (Postfix) with ESMTP id 84C1C300359 for ; Fri, 1 May 2015 10:29:33 -0500 (CDT) X-Amavis-Modified: Mail body modified (using disclaimer) - mail.int.journyx.com X-Virus-Scanned: amavisd-new at mail.int.journyx.com Received: from mail.int.journyx.com ([127.0.0.1]) by localhost (mail.int.journyx.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ozdD-GJUJddu for ; Fri, 1 May 2015 10:29:33 -0500 (CDT) Received: from mail.int.journyx.com (zimbra.int.journyx.com [192.168.100.5]) by mail.int.journyx.com (Postfix) with ESMTP id 39F80300357 for ; Fri, 1 May 2015 10:29:33 -0500 (CDT) Date: Fri, 1 May 2015 10:29:33 -0500 (CDT) From: Scott Whitney Reply-To: Scott Whitney To: pgsql-admin@postgresql.org Message-ID: <1652559419.2123167.1430494173076.JavaMail.zimbra@mail.int.journyx.com> In-Reply-To: References: Subject: Re: pg_dump and pg_restore with multiple streams does Not seem to improve overall times MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2123166_1457247175.1430494173075" X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC42 (Win)/8.6.0_GA_1153) Thread-Topic: pg_dump and pg_restore with multiple streams does Not seem to improve overall times Thread-Index: kYLcaFXZqu4iQjEtznJGWNGs+NtoSw== X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org ------=_Part_2123166_1457247175.1430494173075 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable A few years back, I wrote something very similar to what ended up being in = PG, and I've been using it successfully ever since.=20 I did find there's a law of diminishing returns after about 6 threads runni= ng pg_dump.=20 In my case, I'm on a 6-drive SSD RAID and getting nowhere near the IOPS of = the RAID,=20 but my suspicion is either that I'm maxing my RAID controller's potential o= r pg_dump just=20 won't get any faster.=20 16 cores 3.4GHz in my case.=20 Now, the interesting thing is that I tried this on spindle arrays, single d= rives, multiple machines (quad core,=20 quad proc, dual core, dual core-dual proc, etc) and there always seemed to = be a diminishing return around=20 6 or 8 jobs running.=20 ----- On May 1, 2015, at 10:22 AM, Scott Ribe = wrote:=20 > On May 1, 2015, at 9:16 AM, Susan K. McClure wrote: > > The DB in question is ~25GB. The processor has 24 Cpus, 12 cores >> I have tried with "--jobs =3D 8, 12, and 20" with little or no discernib= le > > improvements. > I would immediately suspect that you might be disk-bound instead of CPU-b= ound. > What=E2=80=99s the setup of your storage system? > -- > Scott Ribe > scott_ribe@elevated-dev.com > http://www.elevated-dev.com/ > https://www.linkedin.com/in/scottribe/ > (303) 722-0567 voice > -- > Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-admin Journyx, Inc.=0A 7600 Burnet Road #300=0A Austin, TX 78757=0A www.journyx.com=0A =0A p 512.834.8888=0A f 512.834.8858=0A Do you receive our promotional emails? You can subscribe or unsubscribe to those emails at http://go.journyx.com/emailPreference/e/4932/714=0A ------=_Part_2123166_1457247175.1430494173075 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
A few years back, I wrote something very si= milar to what ended up being in PG, and I've been using it successfully eve= r since.

I did find there's a= law of diminishing returns after about 6 threads running pg_dump.

In my case, I'm on a 6-drive SSD RAID= and getting nowhere near the IOPS of the RAID,
but my suspicion = is either that I'm maxing my RAID controller's potential or pg_dump just
won't get any faster.

16 cores 3.4GHz in my case.

= Now, the interesting thing is that I tried this on spindle arrays, single d= rives, multiple machines (quad core,
quad proc, dual core, dual c= ore-dual proc, etc) and there always seemed to be a diminishing return arou= nd
6 or 8 jobs running.

----- On May 1, 2015, at 10:22 AM, Scott Ribe <= ;scott_ribe@elevated-dev.com> wrote:
On May 1, 2015, at 9:16 AM, Susan K. McClure <smcclure@rice.e= du> wrote:
>
> The DB in question is ~25GB.  The proce= ssor has 24 Cpus, 12 cores
>  
> I have tried with "--job= s =3D 8, 12, and 20"  with little or no discernible improvements.
=
I would immediately suspect that you might be disk-bound instead of CPU= -bound. What=E2=80=99s the setup of your storage system?

--
Scot= t Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
ht= tps://www.linkedin.com/in/scottribe/
(303) 722-0567 voice






--
Sent via pgsql-admin mailing list (pgsql-admin@post= gresql.org)
To make changes to your subscription:
http://www.postgres= ql.org/mailpref/pgsql-admin

=


Journyx, Inc.
7600 Burnet Road #300
Austin, TX 78757
www.journyx.com

p 512.834.8888 
f 512-834-8858 

Do you receive our promotional emails? You can subscribe or unsubscribe to those emails here. 

=0A
= ------=_Part_2123166_1457247175.1430494173075--