X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 6BC2932B2B3 for ; Fri, 22 Oct 2004 06:01:34 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 85741-02 for ; Fri, 22 Oct 2004 05:01:15 +0000 (GMT) Received: from mpls-qmqp-03.inet.qwest.net (mpls-qmqp-03.inet.qwest.net [63.231.195.114]) by svr1.postgresql.org (Postfix) with SMTP id 0DC0232B2A9 for ; Fri, 22 Oct 2004 06:01:18 +0100 (BST) Received: (qmail 61471 invoked by uid 0); 22 Oct 2004 05:01:19 -0000 Received: from mpls-pop-14.inet.qwest.net (63.231.195.14) by mpls-qmqp-03.inet.qwest.net with QMQP; 22 Oct 2004 05:01:19 -0000 Received: from 63-227-127-37.dnvr.qwest.net (HELO ?10.0.0.2?) (63.227.127.37) by mpls-pop-14.inet.qwest.net with SMTP; 22 Oct 2004 05:01:18 -0000 Date: Thu, 21 Oct 2004 23:03:34 -0600 Message-Id: <1098421413.21035.78.camel@localhost.localdomain> From: "Scott Marlowe" To: "Joshua Marsh" Cc: pgsql-performance@postgresql.org Subject: Re: Large Database Performance suggestions In-Reply-To: <38242de904102120143c55ba34@mail.gmail.com> References: <38242de904102120143c55ba34@mail.gmail.com> Content-Type: text/plain Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 (1.4.6-2) Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200410/377 X-Sequence-Number: 8853 On Thu, 2004-10-21 at 21:14, Joshua Marsh wrote: > Hello everyone, > > I am currently working on a data project that uses PostgreSQL > extensively to store, manage and maintain the data. We haven't had > any problems regarding database size until recently. The three major > tables we use never get bigger than 10 million records. With this > size, we can do things like storing the indexes or even the tables in > memory to allow faster access. > > Recently, we have found customers who are wanting to use our service > with data files between 100 million and 300 million records. At that > size, each of the three major tables will hold between 150 million and > 700 million records. At this size, I can't expect it to run queries > in 10-15 seconds (what we can do with 10 million records), but would > prefer to keep them all under a minute. > > We did some original testing and with a server with 8GB or RAM and > found we can do operations on data file up to 50 million fairly well, > but performance drop dramatically after that. Does anyone have any > suggestions on a good way to improve performance for these extra large > tables? Things that have come to mind are Replication and Beowulf > clusters, but from what I have recently studied, these don't do so wel > with singular processes. We will have parallel process running, but > it's more important that the speed of each process be faster than > several parallel processes at once. I'd assume that what's happening is that up to a certain data set size, it all fits in memory, and you're going from CPU/memory bandwidth limited to I/O limited. If this is the case, then a faster storage subsystem is the only real answer. If the database is mostly read, then a large RAID5 or RAID 1+0 array should help quite a bit. You might wanna post some explain analyze of the queries that are going slower at some point in size, along with schema for those tables etc...