X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id AA9ACD1B50B for ; Thu, 13 Nov 2003 06:03:34 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 66527-08 for ; Thu, 13 Nov 2003 02:03:04 -0400 (AST) Received: from node-4024d1a2.mdw.onnet.us.uu.net (nsit-s227-70.uchicago.edu [128.135.227.70]) by svr1.postgresql.org (Postfix) with ESMTP id C2DE8D1B535 for ; Thu, 13 Nov 2003 02:02:46 -0400 (AST) Received: from localhost (localhost [127.0.0.1]) by node-4024d1a2.mdw.onnet.us.uu.net (Postfix) with ESMTP id 1E96A37D7E; Wed, 12 Nov 2003 23:58:45 -0600 (CST) Subject: Re: performance optimzations From: Suchandra Thapa To: Rod Taylor Cc: Postgresql Performance In-Reply-To: <1068698132.31569.46.camel@jester> References: <1068658430.1071.42.camel@hepcat> <1068661380.30452.17.camel@jester> <1068663070.5193.21.camel@hepcat> <1068698132.31569.46.camel@jester> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-BcogzFHj1OVm6xatBeRz" Organization: Message-Id: <1068703124.12978.27.camel@hepcat> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.2 (1.2.2-5) Date: 12 Nov 2003 23:58:45 -0600 X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200311/83 X-Sequence-Number: 4643 --=-BcogzFHj1OVm6xatBeRz Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Wed, 2003-11-12 at 22:35, Rod Taylor wrote: > You may want to consider keeping the compressed email in a separate > table than the information describing it. It would mean descriptive > information is more likely to be in RAM, where the body probably doesn't > matter as much (you view them 1 at a time, subjects tend to be listed > all at once). Thanks for the suggestions. Splitting the load between several machines was the original intent of moving the storage from the file system to a database. I believe the schema I'm already using splits out the body due to the size of some attachments. Luckily the code already gzips the email body and abbreviates common email headers so storing compressed emails isn't a problem.=20=20 > Most clients will be interested in say the last 7 days worth of data?=20 > Great.. Start out with 4GB ram on a good Dual CPU -- Opterons seem to > work quite well -- and make sure the motherboard can hold double that in > memory for an upgrade sometime next year when you've become popular. Unfortunately, the hardware available is pretty much fixed in regards to the system. I can play around with the raid configurations and have some limited choice in regards to the raid controller and number of drivers but that's about all in terms of hardware.=20=20=20=20 > I firmly believe lots of RAM is the answer to most IO issues until you > start getting into large sets of active data (>50GB). 64GB ram is fairly > cheap compared to ongoing maintenance of the 30+ drive system required > to get decent throughput. The current file system holding the user and email information indicates the current data has about 64GB (70K accounts, I'm not sure how many are active but 50% might be good guess). This seems to be somewhat of a steady state however. --=20 Suchandra Thapa --=-BcogzFHj1OVm6xatBeRz Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQA/sx2U6nShCjt5AZIRAmVjAKCGAeAFlKcKMrY7sULbwNwoiy59rQCfcqki Ui3n6yqsdei2GQCOzAQOi9o= =q+x/ -----END PGP SIGNATURE----- --=-BcogzFHj1OVm6xatBeRz--