pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed

From: Stephen Frost <sfrost@snowman.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tatsuo Ishii <ishii@postgresql.org>
Cc: pgsql-performance@postgresql.org
Cc: pgsql-hackers@postgresql.org
Subject: Re: pg_dump and thousands of schemas
Date: Thu, 31 May 2012 00:01:49 -0400
Message-ID: <20120531040149.GC1267@tamriel.snowman.net> (raw)
In-Reply-To: <26386.1338436470@sss.pgh.pa.us>
References: <20120529.185149.1127748531209576379.t-ishii@sraoss.co.jp>
	<12147.1338306742@sss.pgh.pa.us>
	<20120530.180620.600165924826262795.t-ishii@sraoss.co.jp>
	<20120531.092043.1953698679173443322.t-ishii@sraoss.co.jp>
	<26386.1338436470@sss.pgh.pa.us>

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Tatsuo Ishii <ishii@postgresql.org> writes:
> > Shall I commit to master and all supported branches?
> 
> I'm not excited by this patch.  It dodges the O(N^2) lock behavior for
> the initial phase of acquiring the locks, but it does nothing for the
> lock-related slowdown occurring in all pg_dump's subsequent commands.
> I think we really need to get in the server-side fix that Jeff Janes is
> working on, and then re-measure to see if something like this is still
> worth the trouble.  I am also a tad concerned about whether we might not
> have problems with parsing memory usage, or some such, with thousands of
> tables being listed in a single command.

I can't imagine a case where it's actually better to incur the latency
penalty (which is apparently on the order of *minutes* of additional
time here..) than to worry about the potential memory usage of having to
parse such a command.

If that's really a concern, where is that threshold, and could we simply
cap pg_dump's operations based on it?  Is 1000 alright?  Doing a 'lock'
w/ 1000 tables at a time is still going to be hugely better than doing
them individually and the amount of gain between every-1000 and
all-at-once is likely to be pretty minimal anyway...

The current situation where the client-to-server latency accounts for
multiple minutes of time is just ridiculous, however, so I feel we need
some form of this patch, even if the server side is magically made much
faster.  The constant back-and-forth isn't cheap.

	Thanks,

		Stephen

Attachments:

  signature.asc (application/pgp-signature, 197B) download

view thread (66+ messages)  latest in thread

Message-ID: <20120531040149.GC1267@tamriel.snowman.net>
Permalink:  ../20120531040149.GC1267@tamriel.snowman.net/
Also on:    postgresql.org/message-id/20120531040149.GC1267@tamriel.snowman.net

copy link · copy postgr.es

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: sfrost@snowman.net, tgl@sss.pgh.pa.us, ishii@postgresql.org, pgsql-hackers@postgresql.org
  Subject: Re: pg_dump and thousands of schemas
  In-Reply-To: <20120531040149.GC1267@tamriel.snowman.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox