Received: from malur.postgresql.org ([217.196.149.56])
 by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from
 <pgsql-admin-owner+M41213=pgsql+2Dadmin=arkaria.postgresql.org@postgresql.org>)
 id 1Uo3FS-0007Ev-97
 for pgsql-admin@arkaria.postgresql.org; Sun, 16 Jun 2013 03:03:22 +0000
Received: from localhost ([127.0.0.1] helo=postgresql.org)
 by malur.postgresql.org with smtp (Exim 4.80) (envelope-from
 <pgsql-admin-owner+M41213=pgsql+2Dadmin=arkaria.postgresql.org@postgresql.org>)
 id 1Uo3FR-0003cc-Dm
 for pgsql-admin@arkaria.postgresql.org; Sun, 16 Jun 2013 03:03:21 +0000
Received: from makus.postgresql.org ([2001:4800:7903:4::125])
 by malur.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <bruce@momjian.us>) id 1Uo3FP-0003cT-2t
 for pgsql-admin@postgresql.org; Sun, 16 Jun 2013 03:03:19 +0000
Received: from momjian.us ([72.94.173.45])
 by makus.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <bruce@momjian.us>) id 1Uo3FK-0002hw-JN
 for pgsql-admin@postgresql.org; Sun, 16 Jun 2013 03:03:17 +0000
Received: from bruce by momjian.us with local (Exim 4.72)
 (envelope-from <bruce@momjian.us>)
 id 1Uo3FJ-0006Cz-9y; Sat, 15 Jun 2013 23:03:13 -0400
Date: Sat, 15 Jun 2013 23:03:13 -0400
From: Bruce Momjian <bruce@momjian.us>
To: Lonni J Friedman <netllama@gmail.com>
Cc: "pgsql-admin@postgresql.org" <pgsql-admin@postgresql.org>
Subject: Re: pg_upgrade with -j shows no perf improvement
Message-ID: <20130616030313.GA22904@momjian.us>
References: 
 <CAP=oouF=fAspj-mzFRj0_3bJh9-iNSgfFfc2ONPgE+Y-a5PF7w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAP=oouF=fAspj-mzFRj0_3bJh9-iNSgfFfc2ONPgE+Y-a5PF7w@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Pg-Spam-Score: 0.5 (/)
List-Archive: <http://archives.postgresql.org/pgsql-admin>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-ID: <pgsql-admin.postgresql.org>
List-Owner: <mailto:pgsql-admin-owner@postgresql.org>
List-Post: <mailto:pgsql-admin@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-admin>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-admin>
X-Mailing-List: pgsql-admin
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

On Fri, Jun 14, 2013 at 02:29:24PM -0700, Lonni J Friedman wrote:
> Greetings,
> I'm in the early stages of preparing to upgrade a production 9.2
> cluster to 9.3, by testing the beta of 9.3.  All of my testing is
> happening on RHEL6-x86_64 on a dedicated server with 128GB RAM and 2x
> Intel Xeon E5-2670 CPUs, with all of $PGDATA residing on an 8 disk
> RAID10 array.
> 
> Currently, a full pg_basebackup of my data is approaching 800GB in
> size (uncompressed), so this isn't a tiny, trivial database.
> 
> I was curious about how much of a performance gain I'd get from
> upgrading with the new -j option to pg_upgrade, so first I performed
> the upgrade without it to get a baseline.  The command I ran for the
> upgrade is as follows:
> time pg_upgrade -v -d /var/lib/pgsql/9.2/data -D
> /var/lib/pgsql/9.3/data -b /usr/pgsql-9.2/bin -B /usr/pgsql-9.3/bin
> 
> time reported the following afterward the upgrade had completed successfully:
> real    24m59.255s
> user    0m17.069s
> sys    15m25.153s
> 
> 
> I then repeated the upgrade (after blowing away $PGDATA, and running
> initdb again for 9.3), and re-ran pg_upgrade with the same command as
> above, only with '-j4' appended to the end.  Surprisingly, the
> completion time was less than 30 seconds faster.  I repeated a third
> time with '-j8', and that was about the same completion time as with
> '-j4'.  I guess I could repeat with 'j2', but I'd be surprised if it
> was dramatically faster when -j4 was only marginally so.  It seems
> like the parallelism of the -j option doesn't seem to be helping much
> at all, in my case.
> 
> Is this expected, or is it possible that there's a bug somewhere?  Let
> me know if I can provide any logs from the upgrade.

The documentation states:

     The <option>--jobs</> option allows multiple CPU cores to be used
     for copying/linking of files and to dump and reload database schemas
     in parallel;  a good place to start is the maximum of the number of
     CPU cores and tablespaces.  This option can dramatically reduce the
     time to upgrade a multi-database server running on a multiprocessor
     machine.

My guess is that you didn't have many tablespaces or databases, or the
copy time overwhelmed the performance improvement of the parallelism.
I am not surprised you didn't see a big win.  Can you test --link
mode?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin