X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id 88D649DC8E1 for ; Tue, 24 Jan 2006 12:41:16 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 41583-04 for ; Tue, 24 Jan 2006 12:41:15 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from hosting.commandprompt.com (128.commandprompt.com [207.173.200.128]) by postgresql.org (Postfix) with ESMTP id 509009DC841 for ; Tue, 24 Jan 2006 12:41:14 -0400 (AST) Received: from [192.168.1.101] (or-67-76-146-141.sta.sprint-hsd.net [67.76.146.141]) (authenticated bits=0) by hosting.commandprompt.com (8.13.4/8.13.4) with ESMTP id k0OGVRFo029978; Tue, 24 Jan 2006 08:31:28 -0800 Message-ID: <43D658D3.5070808@commandprompt.com> Date: Tue, 24 Jan 2006 08:41:55 -0800 From: "Joshua D. Drake" User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: Brad Nicholson CC: pgsql-performance@postgresql.org Subject: Re: Investigating IO Saturation References: <43D65736.9020905@ca.afilias.info> In-Reply-To: <43D65736.9020905@ca.afilias.info> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeded SMTP AUTH authentication, not delayed by milter-greylist-1.6 (hosting.commandprompt.com [192.168.1.101]); Tue, 24 Jan 2006 08:31:28 -0800 (PST) X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.078 required=5 tests=[AWL=0.078] X-Spam-Score: 0.078 X-Spam-Level: X-Archive-Number: 200601/418 X-Sequence-Number: 16896 Brad Nicholson wrote: > I'm investigating a potential IO issue. We're running 7.4 on AIX > 5.1. During periods of high activity (reads, writes, and vacuums), we > are seeing iostat reporting 100% disk usage. I have a feeling that > the iostat numbers are misleading. I can make iostat usage jump from > less than 10% to greater than 95% by running a single vacuum against a > moderate sized table (no noticeable change in the other activity). > Well that isn't surprising. Vacuum is brutal especially on 7.4 as that is pre background writer. What type of IO do you have available (RAID, SCSI?) > Do I actually have a problem with IO? Whether I do or not, at what > point should I start to be concerned about IO problems? If my > understanding is correct, it should be based on the wait time. Here's > the output of vmstat during heavy load (reads, writes, several daily > vacuums and a nightly pg_dump). Wait times appear to be alright, but > my understanding of when to start being concerned about IO starvation > is foggy at best. > > vmstat 5 > kthr memory page faults cpu > ----- ----------- ------------------------ ------------ ----------- > r b avm fre re pi po fr sr cy in sy cs us sy id wa > 2 2 1548418 67130 0 0 0 141 99 0 1295 22784 13128 11 4 71 14 > 3 3 1548422 66754 0 0 0 2127 2965 0 2836 29981 25091 26 4 39 31 > 2 3 1548423 66908 0 0 0 2369 3221 0 3130 34725 28424 25 7 38 30 > 3 5 1548423 67029 0 0 0 2223 3097 0 2722 31885 25929 26 9 33 32 > 3 3 1548423 67066 0 0 0 2366 3194 0 2824 43546 36226 30 5 35 31 > 2 4 1548423 67004 0 0 0 2123 3236 0 2662 25756 21841 22 4 39 35 > 2 4 1548957 66277 0 0 0 1928 10322 0 2941 36340 29906 28 6 > 34 33 > 3 5 1549245 66024 0 0 0 2324 14291 0 2872 39413 25615 25 4 > 34 37 > 2 6 1549282 66107 0 0 0 1930 11189 0 2832 72062 32311 26 5 > 32 38 > 2 4 1549526 65855 0 0 0 2375 9278 0 2822 40368 32156 29 5 37 29 > 2 3 1548984 66227 0 0 0 1732 5065 0 2825 39240 30788 26 5 40 30 > 3 4 1549341 66027 0 0 0 2325 6453 0 2790 37567 30509 28 5 37 30 > 2 4 1549377 65789 0 0 0 1633 2731 0 2648 35533 27395 20 5 39 36 > 1 5 1549765 65666 0 0 0 2272 3340 0 2792 43002 34090 26 5 29 40 > 2 3 1549787 65646 0 0 0 1779 2679 0 2596 37446 29184 22 5 37 36 > 2 5 1548985 66263 0 0 0 2077 3086 0 2778 49579 39940 26 9 35 30 > 2 4 1548985 66473 0 0 0 2078 3093 0 2682 23274 18460 22 3 41 34 > 4 3 1548985 66263 0 0 0 2177 3344 0 2734 43029 35536 29 5 38 28 > 1 4 1548985 66491 0 0 0 1978 3215 0 2739 28291 22672 23 4 41 32 > 3 3 1548985 66422 0 0 0 1732 2469 0 2852 71865 30850 28 5 38 29 >