Received: from localhost (maia-5.hub.org [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 0CE979FB2B9 for ; Tue, 3 Jul 2007 14:19:52 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 31896-02 for ; Tue, 3 Jul 2007 14:19:44 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.183]) by postgresql.org (Postfix) with ESMTP id E8C5B9FB1D6 for ; Tue, 3 Jul 2007 14:19:44 -0300 (ADT) Received: by wa-out-1112.google.com with SMTP id k40so3100262wah for ; Tue, 03 Jul 2007 10:19:43 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; b=seN+2Ba92MHFAV34iA4UxMsinrQOAXv/Ay7lAm1IbP4mUKMTs/Jf3Zzn3uecom5hSs4VrZHZ1emywpTUKmVmz2NCfVTH4mZmM7NeKwGDTQVGPhak8TEiugzCyruo0NOuzw0STxACq2lk2Cht/lkv7WhBoTiCeOr6av2OktdYBMg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:mime-version:content-type; b=syXvF6iP0Zl7mIHieG1X2CmhqiqbixDu5brSRfgyyR8RLSiqBBCTqJ84PNRS1Kx2A5e+ajk73ZtCvM36hAqY1WO+a3xbk8HcMLxCH+cVKysMcQwZiMtLJQLULDMLjXsmQRTZXuHx4ND4W9wYYsaivkQGigaHp8+BRRT2/4S9xIw= Received: by 10.114.37.1 with SMTP id k1mr6400284wak.1183483183810; Tue, 03 Jul 2007 10:19:43 -0700 (PDT) Received: by 10.114.123.20 with HTTP; Tue, 3 Jul 2007 10:19:43 -0700 (PDT) Message-ID: <5f33a0cb0707031019n7ffd4929p4b558304d47c10d@mail.gmail.com> Date: Tue, 3 Jul 2007 13:19:43 -0400 From: "Evan Reiser" To: pgsql-admin@postgresql.org Subject: High IOWAIT times, low iops? Can't figure out what the bottleneck is... MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_120767_3877948.1183483183796" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200707/21 X-Sequence-Number: 26002 ------=_Part_120767_3877948.1183483183796 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline I was wondering if you guys have some suggested settings for our server, i think we are not hardware limited but the configureation is set up incorrectly. For some reason our database seems to have trouble handling 5-10+ inserts per second which seems to be a pretty trivial load for this hardware, we're seeing very high %iowait, this is a pretty typical output for #iostat -m 5 avg-cpu: %user %nice %system %iowait %steal %idle 0.97 0.00 0.82 89.69 0.00 8.52 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 85.40 0.18 0.72 0 3 sdb 0.00 0.00 0.00 0 0 sdc 30.43 0.00 0.76 0 3 sdd 93.91 0.71 0.12 3 0 sda = 2x320GB 7200rpm in RAID1 (operating system) sdc = 2x150GB 10krpm in RAID1 (transaction log is on this array) sdd = 6x150GB 10krpm in RAID 10 (database is on the array) we're using ReiserFS on each of the arrays as the file system OS is debian Etch x64 raid controller = 3ware 9650 12port - 256MB cache 8GB RAM, core 2 duo - quad core 2.4ghz 8MB L2 Cache it would seem like the io subsystem is the limiting factor, but i feel like we should be barely hitting a wall, you can see from the example its writing < 2MB/s to the array. If i try to copy a big file (1-2GB) over to the array, the array hits 300MB/s sustained, it can burst write files < 256mb AT 800MB/S Here's some of our settings shared_buffers = 1024MB temp_buffers = 32MB max_prepared_transactions = 50 work_mem = 64MB maintenance_work_mem = 256MB max_stack_depth = 7MB max_fsm_pages = 512000 vacuum_cost_delay = 20ms fsync = on wal_buffers = 1MB checkpoint_segments = 32 effective_cache_size = 4128MB If you guys have any suggestions it would be greatly appreciated -Evan Reiser ------=_Part_120767_3877948.1183483183796 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I was wondering if you guys have some suggested settings for our server, i think we are not hardware limited but the configureation is set up incorrectly.  For some reason our database seems to have trouble handling 5-10+ inserts per second which seems to be a pretty trivial load for this hardware, we're seeing very high %iowait, this is a pretty typical output for #iostat -m 5

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                0.97      0.00    0.82         89.69     0.00      8.52

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda              85.40         0.18         0.72                 0                3
sdb               0.00         0.00         0.00                  0                0
sdc              30.43         0.00         0.76                 0                3
sdd              93.91         0.71         0.12                 3                0




sda = 2x320GB 7200rpm in RAID1 (operating system)
sdc = 2x150GB 10krpm in RAID1    (transaction log is on this array)
sdd = 6x150GB 10krpm in RAID 10 (database is on the array)

we're using ReiserFS on each of the arrays as the file system
OS is debian Etch x64

raid controller = 3ware 9650 12port - 256MB cache

8GB RAM, core 2 duo - quad core 2.4ghz 8MB L2 Cache

it would seem like the io subsystem is the limiting factor, but i feel like we should be barely hitting a wall, you can see from the example its writing < 2MB/s to the array.  If i try to copy a big file (1-2GB) over to the array, the array hits 300MB/s sustained, it can burst write files < 256mb AT 800MB/S

Here's some of our settings


shared_buffers = 1024MB               
temp_buffers = 32MB                  
max_prepared_transactions = 50       
work_mem = 64MB                      
maintenance_work_mem = 256MB         
max_stack_depth = 7MB                  
max_fsm_pages = 512000         

vacuum_cost_delay = 20ms                      

fsync = on       
wal_buffers = 1MB 
checkpoint_segments = 32

effective_cache_size = 4128MB

If you guys have any suggestions it would be greatly appreciated

-Evan Reiser
------=_Part_120767_3877948.1183483183796--