DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=BI0OBYDvVNDYKal9WpnkEj1MSD+krdFwx2GORwDmzwhLIWOaujPxsayev9ctoY1HOL
	RVGpIADr/6ZU1+DouScd1BzvHZMTUWa372jWk/EYk96aRqTrvk/9DVlD/Bv/ZQFgqbw6
	VV23Zs5Pc9kI+3ElOgPuumsE/m9QSTLlbdxm0=
MIME-Version: 1.0
In-Reply-To: <20100420182030.GA53489@mr-paradox.net>
References: <20100420173936.GA50886@mr-paradox.net>
	<h2w603c8f071004201044tbc8ce8fenb846b7643a2b9376@mail.gmail.com>
	<20100420180351.GA51940@mr-paradox.net>
	<w2ld4e11e981004201112v3571f350zaaa0112d1e035097@mail.gmail.com>
	<20100420182030.GA53489@mr-paradox.net>
Date: Tue, 20 Apr 2010 12:23:51 -0600
Message-ID: <j2qdcc563d11004201123u21219445v4a9129ee7d7b520b@mail.gmail.com>
Subject: Re: Very high effective_cache_size == worse performance?
From: Scott Marlowe <scott.marlowe@gmail.com>
To: David Kerr <dmk@mr-paradox.net>
Cc: Nikolas Everett <nik9000@gmail.com>, Robert Haas <robertmhaas@gmail.com>,
	pgsql-performance@postgresql.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 20, 2010 at 12:20 PM, David Kerr <dmk@mr-paradox.net> wrote:
> On Tue, Apr 20, 2010 at 02:12:15PM -0400, Nikolas Everett wrote:
> - On Tue, Apr 20, 2010 at 2:03 PM, David Kerr <dmk@mr-paradox.net> wrote:
> -
> - > that thought occured to me while I was testing this. I ran a vacuumdb=
 -z
> - > on my database during the load and it didn't impact performance at al=
l.
> - >
> - > Incidentally the code is written to work like this :
> - >
> - > while (read X lines in file){
> - > Process those lines.
> - > write lines to DB.
> - > }
> - >
> - > So i would generally expect to get the benefits of the updated static=
is
> - > once the loop ended. no? =A0(would prepared statements affect that po=
ssibly?)
> - >
> - > Also, while I was debugging the problem, I did load a 2nd file into t=
he DB
> - > ontop of one that had been loaded. So the statistics almost certinaly
> - > should
> - > have been decent at that point.
> - >
> - > I did turn on log_min_duration_statement but that caused performance =
to be
> - > unbearable,
> - > but i could turn it on again if it would help.
> - >
> - > Dave
> -
> -
> - You can absolutely use copy if you like but you need to use a non-stand=
ard
> - jdbc driver: =A0kato.iki.fi/sw/db/postgresql/jdbc/copy/. =A0I've used i=
t in the
> - past and it worked.
> -
> - Is the whole thing going in in one transaction? =A0I'm reasonably sure
> - statistics aren't kept for uncommited transactions.
> -
> - For inserts the prepared statements can only help. =A0For selects they =
can
> - hurt because eventually the JDBC driver will turn them into back end
> - prepared statements that are only planned once. =A0The price here is th=
at that
> - plan may not be the best plan for the data that you throw at it.
> -
> - What was log_min_duration_statement logging that it killed performance?
> -
> - --Nik
>
> Good to know about the jdbc-copy. but this is a huge project and the load=
 is
> just one very very tiny component, I don't think we could introduce anyth=
ing
> new to assist that.
>
> It's not all in one tx. I don't have visibility to the code to determine =
how
> it's broken down, but most likely each while loop is a tx.
>
> I set it to log all statements (i.e., =3D 0.). that doubled the load time=
 from
> ~15 to ~30 hours. I could, of course, be more granular if it would be hel=
pful.

So are you logging to the same drive that has pg_xlog and your
data/base directory on this machine?