pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: "Hugo <Nabble>
From: " <hugo.tech@gmail.com>
To: pgsql-performance@postgresql.org
Subject: Re: pg_dump and thousands of schemas
Date: Tue, 31 Jul 2012 22:33:04 -0700 (PDT)
Message-ID: <1343799184312-5718532.post@n5.nabble.com> (raw)
In-Reply-To: <20120613.104525.2089875055094693929.t-ishii@sraoss.co.jp>
References: <20887.1337956883@sss.pgh.pa.us>
	<20120526.101840.762942175522564494.t-ishii@sraoss.co.jp>
	<20120529.185149.1127748531209576379.t-ishii@sraoss.co.jp>
	<12147.1338306742@sss.pgh.pa.us>
	<20120530.180620.600165924826262795.t-ishii@sraoss.co.jp>
	<CAMkU=1zdr7eOEcbopM6c-+zT1aTaWXsTyA_5ZkZ4rgG7EkxMPQ@mail.gmail.com>
	<CAMkU=1xK=VLzyEeqSVrOxpEuoD7A0tOBA2-BRjXBsCNPkJB5Dw@mail.gmail.com>
	<20120612.175425.2118167759512265051.t-ishii@sraoss.co.jp>
	<CAMkU=1yi6f8VKMuwbED=Ogux51Tw-DMQabt580+1vg5_wf-YVA@mail.gmail.com>
	<20120613.104525.2089875055094693929.t-ishii@sraoss.co.jp>

Hi guys,

I just want to let you know that we have created our own solution to dump
and restore our databases. The tool was written in java and the source is
now on Github (MIT license): https://github.com/tig100/JdbcPgBackup

The main approach was to cache all database objects - schemas, tables,
indexes, etc., and instead of having postgres do joins between the
pg_catalog tables (which include thousands of schemas in pg_namespace and
millions of columns in pg_attribute), we do full table scans and then find
which schema or table an object belongs to by looking it up in a hash map in
java, based on schema and table oid's. The dump is not transactionally safe,
so it should be performed on a replica db only (WAL importing disabled), not
on a live db. Some statistics:

Dump 11,000 schemas = 3 hours.
Dump 28,000 schemas = 8 hours.

You can read more about the tool on the github page.

Best regards,
Hugo




--
View this message in context: http://postgresql.1045698.n5.nabble.com/pg-dump-and-thousands-of-schemas-tp5709766p5718532.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.



view thread (66+ messages)  latest in thread

Message-ID: <1343799184312-5718532.post@n5.nabble.com>
Permalink:  ../1343799184312-5718532.post@n5.nabble.com/
Also on:    postgresql.org/message-id/1343799184312-5718532.post@n5.nabble.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: Nabble, hugo.tech@gmail.com
  Subject: Re: pg_dump and thousands of schemas
  In-Reply-To: <1343799184312-5718532.post@n5.nabble.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox