DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=from:to:references:in-reply-to:subject:date:message-id:mime-version
	:content-type:content-transfer-encoding:x-mailer:thread-index
	:content-language;
	b=IyXKd6YzpJK0OnroY4U1oC0mlgjTlcRBj721ag4KA37nZNsy1wrybpx1vFpal7Ayuw
	HkHPz5eSDx0KmHrV9dTl9pnCb11/atszR/6MQCjRNsGQ/J0nk8scrwuj6uirqjKwlM5h
	lu2yznaArMOO9zMChteaUv9qJ4Gc2ItTrp/uM=
From: "mark" <dvlhntr@gmail.com>
To: "'Jehan-Guillaume \(ioguix\) de Rorthais'" <ioguix@free.fr>,
	"'Martin Povolny'" <martin.povolny@solnet.cz>, <pgsql-admin@postgresql.org>
References: <E1PAhuU-0006Fa-DI@ns.solnet.cz>
	<68B59BEDCD36854AADBDF17E91B2937A01A7166E97@EXCHMAIL.staff.internode.com.au>
	<4CC75485.4080901@free.fr>
In-Reply-To: <4CC75485.4080901@free.fr>
Subject: Re: large database: problems with pg_dump and pg_restore
Date: Tue, 26 Oct 2010 20:41:36 -0600
Message-ID: <005a01cb7580$7a9749f0$6fc5ddd0$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: Act1XEoN28UM1wndSGijyJtYzJ8XcAAIM2YQ
Content-Language: en-us

A long time ago, (8.1.11 IIRC)

We got much better speed not using the compression flag with pg_dump =
instead piping to gzip (or better yet something like pbzip2 or pigz, but =
I haven't used them). =20


I think there was a thread about this that had a test case and numbers.=20


IIRC it's because you will further bottleneck a core when using the =
compression flag. Using a pipe the compression can be done on another =
core (or cores if using pbzip2 or pigz) and throughput will be faster.=20


On the restore side hopefully people are now able to use parallel =
restore to improve things when reloading.=20


Just my thoughts,=20


~mark


-----Original Message-----
From: pgsql-admin-owner@postgresql.org =
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Jehan-Guillaume =
(ioguix) de Rorthais
Sent: Tuesday, October 26, 2010 4:22 PM
To: Martin Povolny
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] large database: problems with pg_dump and =
pg_restore

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Or even compress AND split it !
  pg_dump -Fc dbname | split -b 1G - dump_dbname

and restore:
  cat dump_dbname* | pg_restore -d dbname

or
  cat dump_dbname* | pg_restore | psql dbname

Le 26/10/2010 23:51, Samuel Stearns a =C3=A9crit :
> You can also try piping the dump through gzip and then restoring using =
cat:
>=20
> =20
>=20
> pg_dumpall | gzip > db.out-`date +\%Y\%m\%d\%H`.gz
>=20
> =20
>=20
> cat db.out-`date +\%Y\%m\%d\%H`.gz | gunzip | psql template1
>=20
> =20
>=20
> Sam
>=20
> =20
>=20
> =20
>=20
> =20
>=20
> *From:* pgsql-admin-owner@postgresql.org
> [mailto:pgsql-admin-owner@postgresql.org] *On Behalf Of *Martin =
Povolny
> *Sent:* Tuesday, 26 October 2010 10:12 PM
> *To:* pgsql-admin@postgresql.org
> *Subject:* [ADMIN] large database: problems with pg_dump and =
pg_restore
>=20
> =20
>=20
> Hallo,=20
>=20
> =20
>=20
> I have some quite grave problems with dumping and restoring large
> databases (>4GB of dump).
>=20
> I had 5 databases, 4 dumped ok, the 5th, the largest failed dumping: I
> was unable to make a dump in the default 'tar' format. I got this =
message:
>=20
> =20
>=20
> pg_dump: [tar archiver] archive member too large for tar format
>=20
> =20
>=20
> I got over this issue by using the 'custom' format.
>=20
> =20
>=20
> Unfortunately later on I was only able to restore 3 of the 5 databases
> -- any of the 2 dumps that would get over 4GB in the 'tar' format =
would
> fail.
>=20
> =20
>=20
> /var/tmp# ls -l dumps/
>=20
> total 16294020
>=20
> -rw-r--r-- 1 root root  742611968 2010-10-16 20:36 archiv1.dump
>=20
> -rw-r--r-- 1 root root  317352448 2010-10-16 20:37 archiv2.dump
>=20
> -rw-r--r-- 1 root root 1137477632 2010-10-16 20:41 archiv3.dump
>=20
> -rw-r--r-- 1 root root 3712833536 2010- 10-16 20:57 archiv4.dump
>=20
> -rw-r--r-- 1 root root 8735429632 2010-10-16 21:35 archiv5.dump
>=20
> -rw-r--r-- 1 root root 1253325453 2010-10-16 22:59 bb.dump
>=20
> =20
>=20
> archiv1-4 went OK, archiv5 -- the largest in 'tar' format -- failed =
and
> the 'bb.dump' which is in the 'custom' format, failed too.
>=20
> =20
>=20
> I got these messages:
>=20
> =20
>=20
> for the archiv5 in the 'tar' format:=20
>=20
> =20
>=20
> pg_restore: [tar archiv=C3=A1=C5=99] nalezena po=C5=A1kozen=C3=A1 tar =
hlavi=C4=8Dka v STEX
> (p=C5=99edpokl=C3=A1d=C3=A1no 100, vypo=C4=8Dteno 34044) pozice =
souboru 7750193152
>=20
> =20
>=20
> sorry, it's in my native locale, but is says "found a corrupted tar
> header in STEX (expected 100, calculated 34044) file position =
7750193152
>=20
> =20
>=20
> for the bb.dump in the 'custom' format:
>=20
> =20
>=20
> pg_restore: [vlastn=C3=AD archiv=C3=A1=C5=99] unexpected end of file
>=20
> =20
>=20
> 'vlastn=C3=AD archiv =C3=A1=C5=99' is again in my locale, it should be =
in English "own
> archiver"
>=20
> =20
>=20
> Later I tried to utilize the -I and -i switches of pg_restore to =
restore
> data that are in the archive behing the table that was not restored. =
But
> got the same error message.
>=20
> =20
>=20
> The dump was created on postgresql-8.3  8.3.3-1~bpo40+1 from debian
> backports. I was trying to restore on this version and later on using
> postgresql-8.4 8.4.5-1~bpo50+1 from debian backports, finally I tried
> 64bit version of postgresql-8.4 8.4.5-0ubuntu10.10. No change, still =
the
> same error messages.
>=20
> =20
>=20
> I welcome any help and/or hints on this issue as I need to dump and
> restore several large databases.
>=20
> =20
>=20
> Regards,
>=20
>=20
> --
> Mgr. Martin Povoln=C3=BD, soLNet, s.r.o.,
> +420777714458, martin.povolny@solnet.cz
>=20

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzHVIAACgkQxWGfaAgowiL30ACglAXjKXTOZBsmrW5LFZzb8G83
XawAoIVc1UVkW4UQy5lK/jLNARxCb2QN
=3DAR/f
-----END PGP SIGNATURE-----

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin