Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229])
	by mail.postgresql.org (Postfix) with ESMTP id 21C5EDA6299
	for <pgsql-performance@postgresql.org>;
	Wed,  1 Aug 2012 02:33:21 -0300 (ADT)
Received: from sam.nabble.com ([216.139.236.26])
	by magus.postgresql.org with esmtp (Exim 4.72)
	(envelope-from <hugo.tech@gmail.com>) id 1SwRYd-0005wi-6W
	for pgsql-performance@postgresql.org; Wed, 01 Aug 2012 05:33:20 +0000
Received: from [192.168.236.26] (helo=sam.nabble.com)
	by sam.nabble.com with esmtp (Exim 4.72)
	(envelope-from <hugo.tech@gmail.com>) id 1SwRYO-0000er-AH
	for pgsql-performance@postgresql.org; Tue, 31 Jul 2012 22:33:04 -0700
Date: Tue, 31 Jul 2012 22:33:04 -0700 (PDT)
From: "Hugo <Nabble>" <hugo.tech@gmail.com>
To: pgsql-performance@postgresql.org
Message-ID: <1343799184312-5718532.post@n5.nabble.com>
In-Reply-To: <20120613.104525.2089875055094693929.t-ishii@sraoss.co.jp>
References: <20887.1337956883@sss.pgh.pa.us>
	<20120526.101840.762942175522564494.t-ishii@sraoss.co.jp>
	<20120529.185149.1127748531209576379.t-ishii@sraoss.co.jp>
	<12147.1338306742@sss.pgh.pa.us>
	<20120530.180620.600165924826262795.t-ishii@sraoss.co.jp>
	<CAMkU=1zdr7eOEcbopM6c-+zT1aTaWXsTyA_5ZkZ4rgG7EkxMPQ@mail.gmail.com>
	<CAMkU=1xK=VLzyEeqSVrOxpEuoD7A0tOBA2-BRjXBsCNPkJB5Dw@mail.gmail.com>
	<20120612.175425.2118167759512265051.t-ishii@sraoss.co.jp>
	<CAMkU=1yi6f8VKMuwbED=Ogux51Tw-DMQabt580+1vg5_wf-YVA@mail.gmail.com>
	<20120613.104525.2089875055094693929.t-ishii@sraoss.co.jp>
Subject: Re: pg_dump and thousands of schemas
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Pg-Spam-Score: 0.9 (/)
X-Archive-Number: 201208/1
X-Sequence-Number: 47539

Hi guys,

I just want to let you know that we have created our own solution to dump
and restore our databases. The tool was written in java and the source is
now on Github (MIT license): https://github.com/tig100/JdbcPgBackup

The main approach was to cache all database objects - schemas, tables,
indexes, etc., and instead of having postgres do joins between the
pg_catalog tables (which include thousands of schemas in pg_namespace and
millions of columns in pg_attribute), we do full table scans and then find
which schema or table an object belongs to by looking it up in a hash map in
java, based on schema and table oid's. The dump is not transactionally safe,
so it should be performed on a replica db only (WAL importing disabled), not
on a live db. Some statistics:

Dump 11,000 schemas = 3 hours.
Dump 28,000 schemas = 8 hours.

You can read more about the tool on the github page.

Best regards,
Hugo




--
View this message in context: http://postgresql.1045698.n5.nabble.com/pg-dump-and-thousands-of-schemas-tp5709766p5718532.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.