From: Josh Berkus <josh@agliodbs.com>
To: Jack Coates <jack@lyris.com>
To: pgsql-performance@postgresql.org
Subject: Re: tuning questions
Date: Thu, 4 Dec 2003 08:59:06 -0800
Message-ID: <200312040859.06963.josh@agliodbs.com> (raw)
In-Reply-To: <1070553983.6498.52.camel@cletus.lyris.com>
References: <1070553983.6498.52.camel@cletus.lyris.com>
Jack,
> The application is on another server, and does this torture test: it
> builds a large table (~6 million rows in one test, ~18 million in
> another). Rows are then pulled in chunks of 4 to 6 thousand, acted on,
> and inserted back into another table (which will of course eventually
> grow to the full size of the first).
>e tweaked shared buffers to 8192, pushed sort memory to 2048, vacuum
> memory to 8192, and effective cache size to 10000.
> /proc/sys/kernel/shmmax is set to 1600000000 and /proc/sys/fs/file-max
> is set to 65536. Ulimit -n 3192.
Have you read this?
http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html
Actually, your situation is not "worst case". For one thing, your process is
effectively single-user; this allows you to throw all of your resources at
one user. The problem is that your settings have effectively throttled PG
at a level appropriate to a many-user and/or multi-purpose system. You need
to "open them up".
For something involving massive updating/transformation like this, once you've
done the basics (see that URL above) the main settings which will affect you
are sort_mem and checkpoint_segments, both of which I'd advise jacking way up
(test by increments). Raising wal_buffers wouldn't hurt either.
Also, give some thought to running VACUUM and/or ANALYZE between segments of
your procedure. Particularly if you do updates to many rows of a table and
then query based on the changed data, it is vital to run an ANALYZE first,
and usually a good idea to run a VACUUM if it was an UPDATE or DELETE and not
an INSERT.
--
Josh Berkus
Aglio Database Solutions
San Francisco
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: josh@agliodbs.com, jack@lyris.com
Subject: Re: tuning questions
In-Reply-To: <200312040859.06963.josh@agliodbs.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox