From: Ron <rjpeace@earthlink.net>
To: Luke Lonergan <LLonergan@greenplum.com>
To: pgsql-performance@postgresql.org
Subject: Re: Hardware/OS recommendations for large databases
Date: Sun, 27 Nov 2005 12:10:44 -0500
Message-ID: <6.2.5.6.0.20051127114155.01dbf868@earthlink.net> (raw)
In-Reply-To: <3E37B936B592014B978C4415F90D662DE11EBC@MI8NYCMAIL06.Mi8.co m>
References: <3E37B936B592014B978C4415F90D662DE11EBC@MI8NYCMAIL06.Mi8.com>
At 01:18 AM 11/27/2005, Luke Lonergan wrote:
>For data warehousing its pretty well open and shut. To use all cpus
>and io channels on each query you will need mpp.
>
>Has anyone done the math.on the original post? 5TB takes how long
>to scan once? If you want to wait less than a couple of days just
>for a seq scan, you'd better be in the multi-gb per second range.
More than a bit of hyperbole there Luke.
Some common RW scenarios:
Dual 1GbE NICs => 200MBps => 5TB in 5x10^12/2x10^8= 25000secs=
~6hrs57mins. Network stuff like re-transmits of dropped packets can
increase this, so network SLA's are critical.
Dual 10GbE NICs => ~1.6GBps (10GbE NICs can't yet do over ~800MBps
apiece) => 5x10^12/1.6x10^9= 3125secs= ~52mins. SLA's are even
moire critical here.
If you are pushing 5TB around on a regular basis, you are not wasting
your time & money on commodity <= 300MBps RAID HW. You'll be using
800MBps and 1600MBps high end stuff, which means you'll need ~1-2hrs
to sequentially scan 5TB on physical media.
Clever use of RAM can get a 5TB sequential scan down to ~17mins.
Yes, it's a lot of data. But sequential scan times should be in the
mins or low single digit hours, not days. Particularly if you use
RAM to maximum advantage.
Ron
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-performance@postgresql.org
Cc: rjpeace@earthlink.net, LLonergan@greenplum.com
Subject: Re: Hardware/OS recommendations for large databases
In-Reply-To: <6.2.5.6.0.20051127114155.01dbf868@earthlink.net>
References: <3E37B936B592014B978C4415F90D662DE11EBC@MI8NYCMAIL06.Mi8.com> <6.2.5.6.0.20051127114155.01dbf868@earthlink.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox