pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Restore of pg_dump taking a long time...
9+ messages / 7 participants
[nested] [flat]

* Restore of pg_dump taking a long time...
@ 2006-05-24 11:38 Ruairi <rcarroll@bluemetrix.com>
  2006-05-24 18:47 ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 9+ messages in thread

From: Ruairi @ 2006-05-24 11:38 UTC (permalink / raw)
  To: pgsql-admin

Hi,
I'm currently restoring a fairly large DB from a pg_dump and it's taking 
about 12 hours to finish. The main part of this time is spent creating 
indexes. Is there anyway I can speed up the restore process, or do i just 
have to wait? I'm using postgres 8.1.3 on freebsd and the pg_dump is gzip'd. 
I'm restoring with the following command : "zcat <gzip'd db> | psql -U 
postgres <destination db>"

Regards,
Ruairi Carroll 




^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
@ 2006-05-24 18:47 ` Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:42   ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
  0 siblings, 1 reply; 9+ messages in thread

From: Tom Lane @ 2006-05-24 18:47 UTC (permalink / raw)
  To: Ruairi <rcarroll@bluemetrix.com>; +Cc: pgsql-admin

"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking 
> about 12 hours to finish. The main part of this time is spent creating 
> indexes. Is there anyway I can speed up the restore process, or do i just 
> have to wait?

There's not much you can do to improve the performance of a restore
already in progress.  Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.

(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
  2006-05-24 18:47 ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-05-24 19:42   ` Todd A. Cook <tcook@blackducksoftware.com>
  2006-05-24 19:48     ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 9+ messages in thread

From: Todd A. Cook @ 2006-05-24 19:42 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

Tom Lane wrote:
>
> There's not much you can do to improve the performance of a restore
> already in progress.  Next time you might want to think about kicking up
> maintenance_work_mem before you start it; that's about the only thing
> you can really do to speed up index builds.

I have found that increasing maintenance_work_mem can decrease index
build speed on large tables:

maintenance_work_mem    32768    262144     1048576     2097151
---------------------------------------------------------------
index 400M rows (min)      55        61          68          70
index  45M rows (min)    4.03      4.22        4.88        2.55

These timings were done on a machine with 8GB ram and postgresql 8.1.3
(from the RPMs on postgresql.org).  Each time is an average of three
runs; before each run, postgres was shut down and the filesystem with
/var/lib/pgsql was remounted to clear the filesystem cache.

I'm assuming that the times are increasing because there is less
memory available for the filesystem cache.  Similarly, I assume that
the runs on the smaller table with maintenance_work_mem = 2097151
are faster because everything fits in that space.

-- todd



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
  2006-05-24 18:47 ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:42   ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
@ 2006-05-24 19:48     ` Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:56       ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
  0 siblings, 1 reply; 9+ messages in thread

From: Tom Lane @ 2006-05-24 19:48 UTC (permalink / raw)
  To: Todd A. Cook <tcook@blackducksoftware.com>; +Cc: Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

"Todd A. Cook" <tcook@blackducksoftware.com> writes:
> I have found that increasing maintenance_work_mem can decrease index
> build speed on large tables:

You should probably re-measure when 8.2 comes out; we've fixed a number
of performance issues in the sorting code that might cause that.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
  2006-05-24 18:47 ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:42   ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
  2006-05-24 19:48     ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-05-24 19:56       ` Todd A. Cook <tcook@blackducksoftware.com>
  2006-05-25 20:51         ` Re: Restore of pg_dump taking a long time... Jim Nasby <jnasby@pervasive.com>
  0 siblings, 1 reply; 9+ messages in thread

From: Todd A. Cook @ 2006-05-24 19:56 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

Tom Lane wrote:
> "Todd A. Cook" <tcook@blackducksoftware.com> writes:
>> I have found that increasing maintenance_work_mem can decrease index
>> build speed on large tables:
> 
> You should probably re-measure when 8.2 comes out; we've fixed a number
> of performance issues in the sorting code that might cause that.

Thanks.  I'll do that.  If I can, I'll try it sooner on a build
from CVS.

-- todd



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
  2006-05-24 18:47 ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:42   ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
  2006-05-24 19:48     ` Re: Restore of pg_dump taking a long time... Tom Lane <tgl@sss.pgh.pa.us>
  2006-05-24 19:56       ` Re: Restore of pg_dump taking a long time... Todd A. Cook <tcook@blackducksoftware.com>
@ 2006-05-25 20:51         ` Jim Nasby <jnasby@pervasive.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Jim Nasby @ 2006-05-25 20:51 UTC (permalink / raw)
  To: Todd A. Cook <tcook@blackducksoftware.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

On May 24, 2006, at 2:56 PM, Todd A. Cook wrote:

> Tom Lane wrote:
>> "Todd A. Cook" <tcook@blackducksoftware.com> writes:
>>> I have found that increasing maintenance_work_mem can decrease index
>>> build speed on large tables:
>> You should probably re-measure when 8.2 comes out; we've fixed a  
>> number
>> of performance issues in the sorting code that might cause that.
>
> Thanks.  I'll do that.  If I can, I'll try it sooner on a build
> from CVS.

If you'll be messing around with CVS, you might also want to try this  
patch: http://jim.nasby.net/misc/pgsqlcompression/compress-sort.patch  
(which was written by someone else). It hacks compression into the on- 
disk sort code, which has shown a 50% speed improvement on my  
machine. It should be fine to use for loading a database, but you  
wouldn't want to leave it in for serious use (IIRC there's some cases  
it flat-out doesn't handle right now).

You could probably apply that to 8.1.4 as well if you wanted to; it  
should be fine for just loading the database.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461





^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
@ 2006-05-24 18:56 mcelroy, tim <tim.mcelroy@bostonstock.com>
  2006-05-24 19:00 ` Re: Restore of pg_dump taking a long time... Scott Marlowe <smarlowe@g2switchworks.com>
  0 siblings, 1 reply; 9+ messages in thread

From: mcelroy, tim @ 2006-05-24 18:56 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; Ruairi <rcarroll@bluemetrix.com>; +Cc: pgsql-admin

I have found that the following steps have increased the speed of my
restores:
1. dropdb <DBNAME>
2. createdb <DBANME>
3. Increase maintenance_work_mem as Tom mentioned.  I do this at
restore/runtime
4. Increase work_mem as Tom mentioned.  I do this at restore/runtime
5. renice -20 -p <pid #> (on a Linux box)  <= This step can be performed
while the restore is running but you need root privilege

Tim

 -----Original Message-----
From: 	pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org]  On Behalf Of Tom Lane
Sent:	Wednesday, May 24, 2006 2:48 PM
To:	Ruairi
Cc:	pgsql-admin@postgresql.org
Subject:	Re: [ADMIN] Restore of pg_dump taking a long time... 

"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking 
> about 12 hours to finish. The main part of this time is spent creating 
> indexes. Is there anyway I can speed up the restore process, or do i just 
> have to wait?

There's not much you can do to improve the performance of a restore
already in progress.  Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.

(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
  2006-05-24 18:56 Re: Restore of pg_dump taking a long time... mcelroy, tim <tim.mcelroy@bostonstock.com>
@ 2006-05-24 19:00 ` Scott Marlowe <smarlowe@g2switchworks.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Scott Marlowe @ 2006-05-24 19:00 UTC (permalink / raw)
  To: mcelroy, tim <tim.mcelroy@bostonstock.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

On Wed, 2006-05-24 at 13:56, mcelroy, tim wrote:
> I have found that the following steps have increased the speed of my
> restores:
> 1. dropdb <DBNAME>
> 2. createdb <DBANME>
> 3. Increase maintenance_work_mem as Tom mentioned.  I do this at
> restore/runtime
> 4. Increase work_mem as Tom mentioned.  I do this at restore/runtime
> 5. renice -20 -p <pid #> (on a Linux box)  <= This step can be
> performed while the restore is running but you need root privilege

Note that if there's no other data in the database that you need to
worry about, you can also restore with fsync off.  I.e. if a database
cluster could just be re-inited should a power failure occur.

If you've got already important data you can't afford to lose in the
cluster, then do not turn off fsync.

Also, remember to turn it back on before you go into production.



^ permalink  raw  reply  [nested|flat] 9+ messages in thread

* Re: Restore of pg_dump taking a long time...
@ 2006-05-24 20:05 Naomi Walker <nwalker@eldocomp.com>
  0 siblings, 0 replies; 9+ messages in thread

From: Naomi Walker @ 2006-05-24 20:05 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Ruairi <rcarroll@bluemetrix.com>; pgsql-admin

I would typically drop the indicies before taking the dump, and recreate
in the new instance.  I know the loads go alot faster that way.  Perhaps
the index creation would be the same.

Tom Lane wrote:
> "Ruairi" <rcarroll@bluemetrix.com> writes:
>> I'm currently restoring a fairly large DB from a pg_dump and it's taking 
>> about 12 hours to finish. The main part of this time is spent creating 
>> indexes. Is there anyway I can speed up the restore process, or do i just 
>> have to wait?
> 
> There's not much you can do to improve the performance of a restore
> already in progress.  Next time you might want to think about kicking up
> maintenance_work_mem before you start it; that's about the only thing
> you can really do to speed up index builds.
> 
> (Actually, if there are lots of indexes yet to be built, increasing
> the setting in postgresql.conf and SIGHUP'ing the postmaster could
> be worth doing, as that should affect the builds yet to be done.)
> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster


-- 
----------------------------------------------------------------------------
Naomi Walker                          Chief Information Officer
Eldorado Computing, Inc.              nwalker@eldocomp.com 602-604-3100
----------------------------------------------------------------------------
What lies behind us and what lies before us are tiny matters compared to
what lies within us.     - William Morrow
----------------------------------------------------------------------------

-- CONFIDENTIALITY NOTICE --

Information transmitted by thisĀ e-mail is proprietary to MphasiS and/or its Customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that isĀ privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mail from your records.



^ permalink  raw  reply  [nested|flat] 9+ messages in thread


end of thread, other threads:[~2006-05-25 20:51 UTC | newest]

Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-05-24 11:38 Restore of pg_dump taking a long time... Ruairi <rcarroll@bluemetrix.com>
2006-05-24 18:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-05-24 19:42   ` Todd A. Cook <tcook@blackducksoftware.com>
2006-05-24 19:48     ` Tom Lane <tgl@sss.pgh.pa.us>
2006-05-24 19:56       ` Todd A. Cook <tcook@blackducksoftware.com>
2006-05-25 20:51         ` Jim Nasby <jnasby@pervasive.com>
2006-05-24 18:56 Re: Restore of pg_dump taking a long time... mcelroy, tim <tim.mcelroy@bostonstock.com>
2006-05-24 19:00 ` Scott Marlowe <smarlowe@g2switchworks.com>
2006-05-24 20:05 Re: Restore of pg_dump taking a long time... Naomi Walker <nwalker@eldocomp.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox