pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Josh Berkus <josh@agliodbs.com>
To: Jack Coates <jack@lyris.com>
To: pgsql-performance@postgresql.org
Subject: Re: tuning questions
Date: Thu, 4 Dec 2003 08:59:06 -0800
Message-ID: <200312040859.06963.josh@agliodbs.com> (raw)
In-Reply-To: <1070553983.6498.52.camel@cletus.lyris.com>
References: <1070553983.6498.52.camel@cletus.lyris.com>

Jack,

> The application is on another server, and does this torture test: it
> builds a large table (~6 million rows in one test, ~18 million in
> another). Rows are then pulled in chunks of 4 to 6 thousand, acted on,
> and inserted back into another table (which will of course eventually
> grow to the full size of the first).

>e tweaked shared buffers to 8192, pushed sort memory to 2048, vacuum
> memory to 8192, and effective cache size to 10000.
> /proc/sys/kernel/shmmax is set to 1600000000 and /proc/sys/fs/file-max
> is set to 65536. Ulimit -n 3192.

Have you read this?
http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html

Actually, your situation is not "worst case".  For one thing, your process is 
effectively single-user; this allows you to throw all of your resources at 
one user.    The problem is that your settings have effectively throttled PG 
at a level appropriate to a many-user and/or multi-purpose system.  You need 
to "open them up".

For something involving massive updating/transformation like this, once you've 
done the basics (see that URL above) the main settings which will affect you 
are sort_mem and checkpoint_segments, both of which I'd advise jacking way up 
(test by increments).  Raising wal_buffers wouldn't hurt either.

Also, give some thought to running VACUUM and/or ANALYZE between segments of 
your procedure.    Particularly if you do updates to many rows of a table and 
then query based on the changed data, it is vital to run an ANALYZE first, 
and usually a good idea to run a VACUUM if it was an UPDATE or DELETE and not 
an INSERT.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco



view thread (32+ messages)  latest in thread

Message-ID: <200312040859.06963.josh@agliodbs.com>
Permalink:  ../200312040859.06963.josh@agliodbs.com/
Also on:    postgresql.org/message-id/200312040859.06963.josh@agliodbs.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: josh@agliodbs.com, jack@lyris.com
  Subject: Re: tuning questions
  In-Reply-To: <200312040859.06963.josh@agliodbs.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox