From: John Arbash Meinel <john@arbash-meinel.com>
To: Markus Schaber <schabios@logi-track.com>
Cc: Ramon Bastiaans <bastiaans@sara.nl>
Cc: pgsql-performance@postgresql.org
Subject: Re: multi billion row tables: possible or insane?
Date: Tue, 01 Mar 2005 10:44:58 -0600
Message-ID: <42249C0A.1090300@arbash-meinel.com> (raw)
In-Reply-To: <422497C8.4040109@logi-track.com>
References: <42243725.6060804@sara.nl>
<422487E4.6040107@arbash-meinel.com>
<422497C8.4040109@logi-track.com>
Markus Schaber wrote:
>Hi, John,
>
>John Arbash Meinel schrieb:
>
>
>
>>>I am doing research for a project of mine where I need to store
>>>several billion values for a monitoring and historical tracking system
>>>for a big computer system. My currect estimate is that I have to store
>>>(somehow) around 1 billion values each month (possibly more).
>>>
>>>
>>>
>>If you have that 1 billion perfectly distributed over all hours of the
>>day, then you need 1e9/30/24/3600 = 385 transactions per second.
>>
>>
>
>I hope that he does not use one transaction per inserted row.
>
>In your in-house tests, we got a speedup factor of up to some hundred
>when bundling rows on insertions. The fastest speed was with using
>bunches of some thousand rows per transaction, and running about 5
>processes in parallel.
>
>
You're right. I guess it just depends on how the data comes in, and what
you can do at the client ends. That is kind of where I was saying put a
machine in front which gathers up the information, and then does a batch
update. If your client can do this directly, then you have the same
advantage.
>
>
John
=:->
Attachments:
signature.asc (application/pgp-signature, 252B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: john@arbash-meinel.com, schabios@logi-track.com, bastiaans@sara.nl
Subject: Re: multi billion row tables: possible or insane?
In-Reply-To: <42249C0A.1090300@arbash-meinel.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox