Received: from localhost (maia-3.hub.org [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 52EF39FBF97 for ; Thu, 28 Jun 2007 10:18:17 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 38949-08 for ; Thu, 28 Jun 2007 10:17:59 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.233]) by postgresql.org (Postfix) with ESMTP id 170439FB9A0 for ; Thu, 28 Jun 2007 10:18:06 -0300 (ADT) Received: by nz-out-0506.google.com with SMTP id l8so43355nzf for ; Thu, 28 Jun 2007 06:18:03 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Gp22MXY4yc+jcBpUqUp0KZdu9vg6M00HmNJ0eDJxbAWZiN0cI5ZzFhvyrcvWv/RPiWssKugrM3cz1ks6v/7RI6aOs7F+3ltnzKQd39S0vrmP4XfzaTQfVNgzOq/d8+boOzmXkFVDH473GBcrefrovQ6RWoJvIIXw2VCOCMYkQ4E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ox/Bbu4Wp1A5A2jtqGbEzo3CiYJ1Ou9RWQwtinQtQzLHSi6tb8pMMzf36KbI0B7N3RKrweTPX0bsBYegOcp3Itxz9g8jmfWiP557vZ8Qk5gBlWkwV6zJ/vrkRv1r12eBswXQuWh1xn0WlxZ1hwOufe+lqZaV2dsblzXAUnVhoSE= Received: by 10.115.107.1 with SMTP id j1mr1545688wam.1183036679678; Thu, 28 Jun 2007 06:17:59 -0700 (PDT) Received: by 10.114.123.20 with HTTP; Thu, 28 Jun 2007 06:17:59 -0700 (PDT) Message-ID: <5f33a0cb0706280617y3ae9d4a2ud338c076d8bf38f0@mail.gmail.com> Date: Thu, 28 Jun 2007 09:17:59 -0400 From: "Evan Reiser" To: "Richard Huxton" Subject: Re: High IOWAIT times, low iops? Need Help with configuration Cc: pgsql-performance@postgresql.org In-Reply-To: <46836092.8020403@archonet.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <5f33a0cb0706271614q2bd34f58qb87a307f6424fba6@mail.gmail.com> <5f33a0cb0706271738x1cd5dacdpaf925120f7a1130c@mail.gmail.com> <46836092.8020403@archonet.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200706/631 X-Sequence-Number: 25664 we've tried benchmarking the array, the data array can write at 800mb/s for files less than 256mb (raid write cache), after which it can sustain 300mb/s, it seems like it can also handle 6-700 iops when benchmarking. it seems to work as expected outside of postgres, I guess we can look at the drivers, let me know if you guys have any other suggestions, thanks for your help, -evan On 6/28/07, Richard Huxton wrote: > Evan Reiser wrote: > > I was wondering if you guys have some suggested settings for our server, i > > think we are not hardware limited but the configureation is set up > > incorrectly. For some reason our database seems to have trouble handling > > 10+ inserts per second which seems to be a pretty trivial load for this > > hardware, we're seeing very high %iowait, this is a pretty typical output > > for #iostat -m 5 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 0.41 0.00 0.41 96.28 0.00 2.90 > > > > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn > > sda 90.63 0.08 0.56 0 2 > > sdc 0.00 0.00 0.00 0 0 > > sdd 94.09 0.19 1.74 0 8 > > > > > > sda = 2x320GB 7200rpm in RAID1 > > sdc = 2x150GB 10krpm in RAID1 (transaction log is on this array) > > sdd = 6x150GB 10krpm in RAID 10 (database is on the array) > > OK, so no write activity on the transaction log, and hardly any reading > on sdd. Your disks are practically idle, and yet iowait is at 96% - very > strange. > > > raid controller = 3ware 9650 12port - 256MB cache > > > > 8GB RAM, core 2 duo - quad core > > > > it would seem like the io subsystem is the limiting factor, but i feel > like > > we should be barely hitting a wall, you can see from the example its > > writing > > < 2MB/s to the array > > > > Here's some of our settings > > > > shared_buffers = 256MB # min 128kB or > max_connections*16kB > > temp_buffers = 32MB # min 800kB > > max_prepared_transactions = 50 # can be 0 or more > > work_mem = 32MB # min 64kB > > maintenance_work_mem = 32MB # min 1MB > > max_stack_depth = 7MB # min 100kB > > > > max_fsm_pages = 512000 # min max_fsm_relations*16, 6 bytes > > Well, you might want to tweak these, but they're not going to completely > kill your io. > > > fsync = off # turns forced synchronization > > You'll be turning this back on in production, I take it? > > Hmm - ideas > 1. Run a VACUUM FULL on your database(s) and see what happens with your > io then > 2. Test a block copy, something like (but a directory on sdd): > dd if=/dev/zero of=/tmp/empty count=1000000 > That should show an upper limit for your write speed. > 3. Google around and check there aren't any issues with your raid > controller and kernel/driver versions. > > -- > Richard Huxton > Archonet Ltd >