Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 7525A1337B4F for ; Wed, 17 Nov 2010 19:43:44 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 48394-04 for ; Wed, 17 Nov 2010 23:43:38 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.8]) by mail.postgresql.org (Postfix) with ESMTP id 9C17B1337B53 for ; Wed, 17 Nov 2010 19:43:36 -0400 (AST) Received: from [192.168.1.6] (static-71-179-240-8.bltmmd.fios.verizon.net [71.179.240.8]) by mrelayeu.kundenserver.de (node=mrbap1) with ESMTP (Nemesis) id 0MFR0G-1PDLsY0ocu-00EaTs; Thu, 18 Nov 2010 00:43:35 +0100 Message-ID: <4CE468A4.80007@2ndquadrant.com> Date: Wed, 17 Nov 2010 18:43:32 -0500 From: Greg Smith User-Agent: Thunderbird 2.0.0.24 (X11/20100317) MIME-Version: 1.0 To: Eric Comeau CC: Merlin Moncure , pgsql-performance@postgresql.org Subject: Re: How to achieve sustained disk performance of 1.25 GB write for 5 mins References: <35FABCF85D99464FB00BC5123DC2A70A0B7EFC38@s228130hz1ew09.apptix-01.savvis.net> In-Reply-To: <35FABCF85D99464FB00BC5123DC2A70A0B7EFC38@s228130hz1ew09.apptix-01.savvis.net> Content-Type: multipart/alternative; boundary="------------030409020902000301060201" X-Provags-ID: V02:K0:vSrUTrSJn6gUhZfR8VUGzsoUlS25VtcaND86ZVU8M1A Uh7Fdr56WYowLRnaRxJMtexZQBo+tcWlq9Ptcb3RBB8FSo+Nfi zIvsFbD2b/P7Fw/5sIEg2X9Nvq/Tt387XrBslkP/kn4N2ShUgH 6eDr7/4/1WJ+fNvFlb7mPZsHZ9XiMHhS/IWuySs2yLetKV+LiT MpWZhlOzbYEPF9XQGR/qpgMX6PCYdef4qgKohRd1tc= X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201011/310 X-Sequence-Number: 41334 This is a multi-part message in MIME format. --------------030409020902000301060201 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Eric Comeau wrote: > Ideally 1 large file, but it may have to be multiple. We find that if > we send multiple files it just causes the disk to thrash more so we > get better throughput by sending one large file. If it's really one disk, sure. The problem you're facing is that your typical drive controller is going to top out at somewhere between 300 - 500MB/s of sequential writes before it becomes the bottleneck. Above somewhere between 6 and 10 drives attached to one controller on current hardware, adding more to a RAID-0 volume only increases the ability to handle seeks quickly. If you want to try and do this with traditional hard drives, I'd guess you'd need 3 controllers with at least 4 short-stroked drives attached to each to have any hope of hitting 1.25GB/s. Once you do that, you'll run into CPU time as the next bottleneck. At that point, you'll probably need one CPU per controller, all writing out at once, to keep up with your target. The only popular hardware design that comes to mind aimed at this sort of thing was Sun's "Thumper" design, most recently seen in the Sun Fire X4540. That put 8 controllers with 6 disks attached to each, claiming "demonstrated up to 2 GB/sec from disk to network". It will take a design like that, running across multiple controllers, to get what you're looking for on the disk side--presuming everything else keeps up. One of the big SSD-on-PCI-e designs mentioned here already may very well end up being a better choice for you here though, as those aren't going to require quite as much hardware all get wired up. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services and Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books --------------030409020902000301060201 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Eric Comeau wrote:
Ideally 1 large file, but it may have to be multiple. We find that if we send multiple files it just causes the disk to thrash more so we get better throughput by sending one large file.

If it's really one disk, sure.  The problem you're facing is that your typical drive controller is going to top out at somewhere between 300 - 500MB/s of sequential writes before it becomes the bottleneck.  Above somewhere between 6 and 10 drives attached to one controller on current hardware, adding more to a RAID-0 volume only increases the ability to handle seeks quickly.  If you want to try and do this with traditional hard drives, I'd guess you'd need 3 controllers with at least 4 short-stroked drives attached to each to have any hope of hitting 1.25GB/s.  Once you do that, you'll run into CPU time as the next bottleneck.  At that point, you'll probably need one CPU per controller, all writing out at once, to keep up with your target.

The only popular hardware design that comes to mind aimed at this sort of thing was Sun's "Thumper" design, most recently seen in the Sun Fire X4540.  That put 8 controllers with 6 disks attached to each, claiming "demonstrated up to 2 GB/sec from disk to network".  It will take a design like that, running across multiple controllers, to get what you're looking for on the disk side--presuming everything else keeps up.

One of the big SSD-on-PCI-e designs mentioned here already may very well end up being a better choice for you here though, as those aren't going to require quite as much hardware all get wired up.

 
-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services and Support        www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
--------------030409020902000301060201--