From: Tom Lane <tgl@sss.pgh.pa.us>
To: Jeff Janes <jeff.janes@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: Craig James <cjames@emolecules.com>
Cc: Hugo <hugo.tech@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: pg_dump and thousands of schemas
Date: Fri, 25 May 2012 12:56:17 -0400
Message-ID: <24045.1337964977@sss.pgh.pa.us> (raw)
In-Reply-To: <CAMkU=1yjA-xQXnFTQVWa5jq-JPXyRdm5RXQQtADDm4T7vKhHoA@mail.gmail.com>
References: <1337843209018-5709766.post@n5.nabble.com>
<CAFwQ8rdZ3LjiwdDuzNR3vN=OryQ6fzDb08hzeKzNbOzrQXNPbA@mail.gmail.com>
<CAMkU=1zgUUqT6ZPAFAMR6cFx01Bbk+y05Fmwk63PBHOMrBPPoQ@mail.gmail.com>
<20120525035455.GC25444@momjian.us>
<CAMkU=1yjA-xQXnFTQVWa5jq-JPXyRdm5RXQQtADDm4T7vKhHoA@mail.gmail.com>
Jeff Janes <jeff.janes@gmail.com> writes:
> There is an operation in pg_dump which is O(#_of_schemata_in_db *
> #_of_table_in_db), or something like that.
> The attached very crude patch reduces that to
> O(log_of_#_of_schemata_in_db * #_of_table_in_db)
> I was hoping this would be a general improvement. It doesn't seem be.
> But it is a very substantial improvement in the specific case of
> dumping one small schema out of a very large database.
Your test case in
<CAMkU=1zedM4VyLVyLuVmoekUnUXkXfnGPer+3bvPm-A_9CNYSA@mail.gmail.com>
shows pretty conclusively that findNamespace is a time sink for large
numbers of schemas, so that seems worth fixing. I don't like this
patch though: we already have infrastructure for this in pg_dump,
namely buildIndexArray/findObjectByOid, so what we should do is use
that not invent something new. I will go see about doing that.
> It seems like dumping one schema would be better optimized by not
> loading up the entire database catalog, but rather by restricting to
> just that schema at the catalog stage.
The reason pg_dump is not built that way is that considerations like
dump order dependencies are not going to work at all if it only looks
at a subset of the database. Of course, dependency chains involving
objects not dumped might be problematic anyway, but I'd still want it
to do the best it could.
> For dumping entire databases, It looks like the biggest problem is
> going to be LockReassignCurrentOwner in the server. And that doesn't
> seem to be easy to fix, as any change to it to improve pg_dump will
> risk degrading normal use cases.
I didn't try profiling the server side, but pg_dump doesn't use
subtransactions so it's not clear to me why LockReassignCurrentOwner
would get called at all ...
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, jeff.janes@gmail.com, bruce@momjian.us, cjames@emolecules.com, hugo.tech@gmail.com
Subject: Re: pg_dump and thousands of schemas
In-Reply-To: <24045.1337964977@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox