pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Evan Reiser <evan.reiser@gmail.com>
To: Richard Huxton <dev@archonet.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: High IOWAIT times, low iops? Need Help with configuration
Date: Thu, 28 Jun 2007 09:17:59 -0400
Message-ID: <5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com> (raw)
In-Reply-To: <46836092.8020403@archonet.com>
References: <5f33a0cb0706271614q2bd34f58qb87a307f6424fba6@mail.gmail.com>
	<5f33a0cb0706271738x1cd5dacdpaf925120f7a1130c@mail.gmail.com>
	<46836092.8020403@archonet.com>

we've tried benchmarking the array, the data array can write at
800mb/s for files less than 256mb (raid write cache), after which it
can sustain 300mb/s, it seems like it can also handle 6-700 iops when
benchmarking. it seems to work as expected outside of postgres, I
guess we can look at the drivers, let me know if you guys have any
other suggestions, thanks for your help, -evan

On 6/28/07, Richard Huxton <dev@archonet.com> wrote:
> Evan Reiser wrote:
> > I was wondering if you guys have some suggested settings for our server, i
> > think we are not hardware limited but the configureation is set up
> > incorrectly.  For some reason our database seems to have trouble handling
> > 10+ inserts per second which seems to be a pretty trivial load for this
> > hardware, we're seeing very high %iowait, this is a pretty typical output
> > for #iostat -m 5
> >
> > avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >                 0.41    0.00     0.41       96.28       0.00     2.90
> >
> > Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
> > sda              90.63         0.08         0.56          0          2
> > sdc               0.00         0.00         0.00          0          0
> > sdd              94.09         0.19         1.74          0          8
> >
> >
> > sda = 2x320GB 7200rpm in RAID1
> > sdc = 2x150GB 10krpm in RAID1    (transaction log is on this array)
> > sdd = 6x150GB 10krpm in RAID 10 (database is on the array)
>
> OK, so no write activity on the transaction log, and hardly any reading
> on sdd. Your disks are practically idle, and yet iowait is at 96% - very
> strange.
>
> > raid controller = 3ware 9650 12port - 256MB cache
> >
> > 8GB RAM, core 2 duo - quad core
> >
> > it would seem like the io subsystem is the limiting factor, but i feel
> like
> > we should be barely hitting a wall, you can see from the example its
> > writing
> > < 2MB/s to the array
> >
> > Here's some of our settings
> >
> > shared_buffers = 256MB                  # min 128kB or
> max_connections*16kB
> > temp_buffers = 32MB                     # min 800kB
> > max_prepared_transactions = 50          # can be 0 or more
> > work_mem = 32MB                         # min 64kB
> > maintenance_work_mem = 32MB             # min 1MB
> > max_stack_depth = 7MB                   # min 100kB
> >
> > max_fsm_pages = 512000          # min max_fsm_relations*16, 6 bytes
>
> Well, you might want to tweak these, but they're not going to completely
> kill your io.
>
> > fsync = off                             # turns forced synchronization
>
> You'll be turning this back on in production, I take it?
>
> Hmm - ideas
> 1. Run a VACUUM FULL on your database(s) and see what happens with your
> io then
> 2. Test a block copy, something like (but a directory on sdd):
>     dd if=/dev/zero of=/tmp/empty count=1000000
>     That should show an upper limit for your write speed.
> 3. Google around and check there aren't any issues with your raid
> controller and kernel/driver versions.
>
> --
>    Richard Huxton
>    Archonet Ltd
>



view thread (4+ messages)  latest in thread

Message-ID: <5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com>
Permalink:  ../5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com/
Also on:    postgresql.org/message-id/5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: evan.reiser@gmail.com, dev@archonet.com
  Subject: Re: High IOWAIT times, low iops? Need Help with configuration
  In-Reply-To: <5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com>
  References: <5f33a0cb0706271614q2bd34f58qb87a307f6424fba6@mail.gmail.com> <5f33a0cb0706271738x1cd5dacdpaf925120f7a1130c@mail.gmail.com> <46836092.8020403@archonet.com> <5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox