X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 574CB5649C for ; Tue, 1 Mar 2005 16:45:03 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 09818-07 for ; Tue, 1 Mar 2005 16:45:02 +0000 (GMT) Received: from server07.icaen.uiowa.edu (server07.icaen.uiowa.edu [128.255.17.47]) by svr1.postgresql.org (Postfix) with ESMTP id E09D45647A for ; Tue, 1 Mar 2005 16:45:01 +0000 (GMT) Received: from server11.icaen.uiowa.edu (server11.icaen.uiowa.edu [128.255.17.51]) by server07.icaen.uiowa.edu (8.13.2/8.12.9) with ESMTP id j21Giw8f022972; (envelope-from ) Tue, 1 Mar 2005 10:44:58 -0600 (CST) Received: from [192.168.1.13] (65-100-23-249.cdrr.qwest.net [65.100.23.249]) (authenticated user=jfmeinel) by server11.icaen.uiowa.edu (8.13.2/smtp-serv-1.7) with ESMTP id j21Gitfh012855 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256); (envelope-from ) Tue, 1 Mar 2005 10:44:57 -0600 (CST) Message-ID: <42249C0A.1090300@arbash-meinel.com> Date: Tue, 01 Mar 2005 10:44:58 -0600 From: John Arbash Meinel User-Agent: Mozilla Thunderbird 1.0 (Macintosh/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Markus Schaber Cc: Ramon Bastiaans , pgsql-performance@postgresql.org Subject: Re: multi billion row tables: possible or insane? References: <42243725.6060804@sara.nl> <422487E4.6040107@arbash-meinel.com> <422497C8.4040109@logi-track.com> In-Reply-To: <422497C8.4040109@logi-track.com> X-Enigmail-Version: 0.89.6.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig2A284578D637918023E3BEB8" X-Virus-Scanned: ClamAV 0.83/723/Thu Feb 24 05:54:24 2005 clamav-milter version 0.80j on clamav.icaen.uiowa.edu X-Virus-Scanned: ClamAV version 0.83, clamav-milter version 0.83 on clamav.icaen.uiowa.edu X-Virus-Status: Clean X-Virus-Status: Clean X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200503/15 X-Sequence-Number: 10720 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig2A284578D637918023E3BEB8 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Markus Schaber wrote: >Hi, John, > >John Arbash Meinel schrieb: > > > >>>I am doing research for a project of mine where I need to store >>>several billion values for a monitoring and historical tracking system >>>for a big computer system. My currect estimate is that I have to store >>>(somehow) around 1 billion values each month (possibly more). >>> >>> >>> >>If you have that 1 billion perfectly distributed over all hours of the >>day, then you need 1e9/30/24/3600 = 385 transactions per second. >> >> > >I hope that he does not use one transaction per inserted row. > >In your in-house tests, we got a speedup factor of up to some hundred >when bundling rows on insertions. The fastest speed was with using >bunches of some thousand rows per transaction, and running about 5 >processes in parallel. > > You're right. I guess it just depends on how the data comes in, and what you can do at the client ends. That is kind of where I was saying put a machine in front which gathers up the information, and then does a batch update. If your client can do this directly, then you have the same advantage. > > John =:-> --------------enig2A284578D637918023E3BEB8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (Darwin) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCJJwKJdeBCYSNAAMRAv4cAKClpavspRY5NLu2savJTYzTgQwZagCfR0r2 82QqSfMqD4nKOkdB8V4fCGs= =gjzI -----END PGP SIGNATURE----- --------------enig2A284578D637918023E3BEB8--