Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125]) by mail.postgresql.org (Postfix) with ESMTP id 93044F6498E for ; Tue, 24 Jul 2012 11:42:23 -0300 (ADT) Received: from mail.anatec.com ([87.84.168.242]) by makus.postgresql.org with smtp (Exim 4.72) (envelope-from ) id 1StgJZ-0001Sk-N5 for pgsql-performance@postgresql.org; Tue, 24 Jul 2012 14:42:23 +0000 Received: from [192.168.10.57] ([192.168.10.57]) by mail.anatec.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Tue, 24 Jul 2012 15:42:07 +0100 Message-ID: <500EB43F.9030904@anatec.com> Date: Tue, 24 Jul 2012 15:42:07 +0100 From: Ioannis Anagnostopoulos User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20120713 Thunderbird/14.0 MIME-Version: 1.0 To: Craig James CC: pgsql-performance@postgresql.org Subject: Re: Heavy inserts load wile querying... References: <500E9A3B.3070609@shopzeus.com> <500EA19A.4040207@anatec.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------050803030607090300050302" X-OriginalArrivalTime: 24 Jul 2012 14:42:07.0736 (UTC) FILETIME=[800DD780:01CD69AA] X-Pg-Spam-Score: -1.9 (-) X-Archive-Number: 201207/301 X-Sequence-Number: 47482 This is a multi-part message in MIME format. --------------050803030607090300050302 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 24/07/2012 15:30, Craig James wrote: > > > On Tue, Jul 24, 2012 at 6:22 AM, Ioannis Anagnostopoulos > > wrote: > > Hello, > The Postres 9.0 database we use gets about 20K inserts per minute. > As long as you don't query at the same time the database is > copying fine. However long running queries seems to delay so much > the db that the application server buffers the incoming data as it > cannot insert them fast enough. The server has 4 HD. One is used > for archive, past static tables, the second is the index of the > current live tables and the third is the current data. The fourth > is the OS. > > The serve specs are: > Intel(R) Xeon(R) CPU W3520 @ 2.67GHz > 4 cores > 18GB Ram > > Do you think that this work load is high that requires an upgrade > to cluster or RAID 10 to cope with it? > > > You need to learn more about what exactly is your bottleneck ... > memory, CPU, or I/O. That said, I suspect you'd be way better off > with this hardware if you built a single software RAID 10 array and > put everything on it. > > Right now, the backup disk and the OS disk are sitting idle most of > the time. With a RAID10 array, you'd at least double, maybe quadruple > your I/O. And if you added a battery-backed RAID controller, you'd > have a pretty fast system. > > Craig > > > Kind Regards > Yiannis > > -- > Sent via pgsql-performance mailing list > (pgsql-performance@postgresql.org > ) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance > > I can only assume that it is an i/o issue. At last this is what I can read from iostat: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 277.50 0.00 20.00 0.00 2344.00 117.20 0.09 2.25 4.50 9.00 sdb 1.00 0.50 207.50 4.50 45228.00 33.50 213.50 2.40 11.34 4.13 87.50 sdc 0.00 0.00 29.50 0.00 4916.00 0.00 166.64 0.11 3.73 1.36 4.00 sdd 0.00 0.00 4.00 179.50 96.00 3010.00 16.93 141.25 828.77 5.45 100.00 avg-cpu: %user %nice %system %iowait %steal %idle 7.60 0.00 2.08 46.45 0.00 43.87 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 61.50 0.00 28.00 0.00 704.00 25.14 0.04 3.04 1.43 4.00 sdb 2.00 0.00 90.50 162.00 19560.00 2992.00 89.31 78.92 194.26 3.76 95.00 sdc 0.00 0.00 10.50 0.00 2160.00 0.00 205.71 0.02 1.90 1.90 2.00 sdd 0.00 0.00 1.50 318.50 24.00 5347.00 16.78 134.72 572.81 3.12 100.00 Where sdb is the data disk and sdd is the index disk. "Top" hardly reports anything more than 10% per postgress process ever, while when the query is running, these numbers on iostat are consistatnly high. At least I can identify my buffering the moment that index hits 100% util. Is there any other way that I can identify bottlenecks in a more positive way? --------------050803030607090300050302 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
On 24/07/2012 15:30, Craig James wrote:


On Tue, Jul 24, 2012 at 6:22 AM, Ioannis Anagnostopoulos <ioannis@anatec.com> wrote:
Hello,
The Postres 9.0 database we use gets about 20K inserts per minute. As long as you don't query at the same time the database is copying fine. However long running queries seems to delay so much the db that the application server buffers the incoming data as it cannot insert them fast enough. The server has 4 HD. One is used for archive, past static tables, the second is the index of the current live tables and the third is the current data. The fourth is the OS.

The serve specs are:
Intel(R) Xeon(R) CPU           W3520  @ 2.67GHz
4 cores
18GB Ram

Do you think that this work load is high that requires an upgrade to cluster or RAID 10 to cope with it?

You need to learn more about what exactly is your bottleneck ... memory, CPU, or I/O.  That said, I suspect you'd be way better off with this hardware if you built a single software RAID 10 array and put everything on it.

Right now, the backup disk and the OS disk are sitting idle most of the time.  With a RAID10 array, you'd at least double, maybe quadruple your I/O.  And if you added a battery-backed RAID controller, you'd have a pretty fast system.

Craig
 

Kind Regards
Yiannis

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

I can only assume that it is an i/o issue. At last this is what I can read from iostat:


Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   277.50    0.00   20.00     0.00  2344.00   117.20     0.09    2.25   4.50   9.00
sdb               1.00     0.50  207.50    4.50 45228.00    33.50   213.50     2.40   11.34   4.13  87.50
sdc               0.00     0.00   29.50    0.00  4916.00     0.00   166.64     0.11    3.73   1.36   4.00
sdd               0.00     0.00    4.00  179.50    96.00  3010.00    16.93   141.25  828.77   5.45 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.60    0.00    2.08   46.45    0.00   43.87

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    61.50    0.00   28.00     0.00   704.00    25.14     0.04    3.04   1.43   4.00
sdb               2.00     0.00   90.50  162.00 19560.00  2992.00    89.31    78.92  194.26   3.76  95.00
sdc               0.00     0.00   10.50    0.00  2160.00     0.00   205.71     0.02    1.90   1.90   2.00
sdd               0.00     0.00    1.50  318.50    24.00  5347.00    16.78   134.72  572.81   3.12 100.00

Where sdb is the data disk and sdd is the index disk. "Top" hardly reports anything more than 10% per postgress process ever, while when the query is running, these numbers on iostat are consistatnly high. At least I can identify my buffering the moment that index hits 100% util. Is there any other way that I can identify bottlenecks in a more positive way?

--------------050803030607090300050302--