pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Merlin Moncure <mmoncure@gmail.com>
To: Worky Workerson <worky.workerson@gmail.com>
Cc: Markus Schaber <schabi@logix-tt.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Date: Wed, 25 Oct 2006 09:20:02 -0400
Message-ID: <b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com> (raw)
In-Reply-To: <ce4072df0610230840r4b3df653h3144dab06bdb9ed2@mail.gmail.com>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
	<453C8B0D.4070803@logix-tt.com>
	<ce4072df0610230840r4b3df653h3144dab06bdb9ed2@mail.gmail.com>

On 10/23/06, Worky Workerson <worky.workerson@gmail.com> wrote:
> The disk load is where I start to get a little fuzzy, as I haven't
> played with iostat to figure what is "normal".  The local drives
> contain PG_DATA as well as all the log files, but there is a
> tablespace on the FibreChannel SAN that contains the destination
> table.  The disk usage pattern that I see is that there is a ton of
> consistent activity on the local disk, with iostat reporting an
> average of 30K Blk_wrtn/s, which I assume is the log files.  Every
> several seconds there is a massive burst of activity on the FC
> partition, to the tune of 250K Blk_wrtn/s.
>
> > On a table with no indices, triggers and contstraints, we managed to
> > COPY about 7-8 megabytes/second with psql over our 100 MBit network, so
> > here the network was the bottleneck.

I'm guessing the high bursts are checkpoints.  Can you check your log
files for pg and see if you are getting warnings about checkpoint
frequency?   You can get some mileage here by increasing wal files.

Have you determined that pg is not swapping?  try upping maintenance_work_mem.

What exactly is your architecture?  is your database server direct
attached to the san? if so, 2gb/4gb fc?  what san?  have you bonnie++
the san?  basically, you can measure iowait to see if pg is waiting on
your disks.

regarding perl, imo the language performance is really about which
libraries you use. the language itself is plenty fast.

merlin



view thread (52+ messages)  latest in thread

Message-ID: <b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com>
Permalink:  ../b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com/
Also on:    postgresql.org/message-id/b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: mmoncure@gmail.com, worky.workerson@gmail.com, schabi@logix-tt.com
  Subject: Re: Best COPY Performance
  In-Reply-To: <b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com>
  References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com> <453C8B0D.4070803@logix-tt.com> <ce4072df0610230840r4b3df653h3144dab06bdb9ed2@mail.gmail.com> <b42b73150610250620r37cf9269m7b6e43b9835bdbcf@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox