X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [64.117.224.130]) by svr1.postgresql.org (Postfix) with ESMTP id 77F27D1B4FD for ; Wed, 10 Sep 2003 00:13:30 +0000 (GMT) Received: from svr1.postgresql.org ([64.117.224.193]) by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024) with ESMTP id 14214-06 for ; Tue, 9 Sep 2003 21:13:01 -0300 (ADT) Received: from davinci.ethosmedia.com (unknown [209.128.84.228]) by svr1.postgresql.org (Postfix) with ESMTP id 0F6D2D1B4E7 for ; Tue, 9 Sep 2003 21:12:58 -0300 (ADT) Received: from [66.219.92.2] (HELO temoku) by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.0.2) with ESMTP id 3625468; Tue, 09 Sep 2003 17:13:28 -0700 Content-Type: text/plain; charset="iso-8859-1" From: Josh Berkus Reply-To: josh@agliodbs.com Organization: Aglio Database Solutions To: Chris Huston , pgsql-performance@postgresql.org Subject: Re: Reading data in bulk - help? Date: Tue, 9 Sep 2003 17:11:32 -0700 User-Agent: KMail/1.4.3 References: <305D78CA-E320-11D7-A142-000393011B1A@bangjafwac.com> In-Reply-To: <305D78CA-E320-11D7-A142-000393011B1A@bangjafwac.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Message-Id: <200309091711.32275.josh@agliodbs.com> X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200309/162 X-Sequence-Number: 3528 Chris, > I've got an application that needs to chunk through ~2GB of data. The=20 > data is ~7000 different sets of 300 records each. I put all of the data= =20 > into a postgres database but that doesn't look like its going to work=20 > because of how the data lives on the disk. Your problem is curable through 4 steps: 1) adjust your postgresql.conf to appropriate levels for memory usage. 2) if those sets of 300 are blocks in some contiguous order, then cluster t= hem=20 to force their physical ordering on disk to be the same order you want to= =20 read them in. This will require you to re-cluster whenever you change a= =20 significant number of records, but from the sound of it that happens in=20 batches. 3) Get better disks, preferrably a RAID array, or just very fast scsi if th= e=20 database is small. If you're budget-constrained, Linux software raid (or= =20 BSD raid) on IDE disks is cheap. What kind of RAID depends on what else=20 you'll be doing with the app; RAID 5 is better for read-only access, RAID 1= +0=20 is better for read-write. 4) Make sure that you aren't dumping the data to the same disk postgreSQL= =20 lives on! Preferably, make sure that your swap partition is on a differen= t=20 disk/array from postgresql. If the computing app is complex and requires= =20 disk reads aside from postgres data, you should make sure that it lives on= =20 yet another disk. Or you can simplify this with a good, really large=20 multi-channel RAID array. --=20 -Josh Berkus Aglio Database Solutions San Francisco