pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Markus Schaber <schabios@logi-track.com>
To: John Arbash Meinel <john@arbash-meinel.com>
Cc: Ramon Bastiaans <bastiaans@sara.nl>
Cc: pgsql-performance@postgresql.org
Subject: Re: multi billion row tables: possible or insane?
Date: Tue, 01 Mar 2005 17:26:48 +0100
Message-ID: <422497C8.4040109@logi-track.com> (raw)
In-Reply-To: <422487E4.6040107@arbash-meinel.com>
References: <42243725.6060804@sara.nl>
	<422487E4.6040107@arbash-meinel.com>

Hi, John,

John Arbash Meinel schrieb:

>> I am doing research for a project of mine where I need to store
>> several billion values for a monitoring and historical tracking system
>> for a big computer system. My currect estimate is that I have to store
>> (somehow) around 1 billion values each month (possibly more).
>>
> If you have that 1 billion perfectly distributed over all hours of the
> day, then you need 1e9/30/24/3600 = 385 transactions per second.

I hope that he does not use one transaction per inserted row.

In your in-house tests, we got a speedup factor of up to some hundred
when bundling rows on insertions. The fastest speed was with using
bunches of some thousand rows per transaction, and running about 5
processes in parallel.

Regard the usual performance tips: Use a small, but fast-writing RAID
for transaction log (no RAID-5 or RAID-6 variants), possibly a mirroring
of two harddisk-backed SSD. Use different disks for the acutal data
(here, LVM2 with growing volumes could be very handy). Have enough RAM.
Use a fast file system.

BTW, as you read about the difficulties that you'll face with this
enormous amount of data: Don't think that your task will much be easier
or cheaper using any other DBMS, neither commercial nor open source. For
all of them, you'll need "big iron" hardware, and a skilled team of
admins to set up and maintain the database.

Markus

-- 
markus schaber | dipl. informatiker
logi-track ag | rennweg 14-16 | ch 8001 zürich
phone +41-43-888 62 52 | fax +41-43-888 62 53
mailto:schabios@logi-track.com | www.logi-track.com



view thread (15+ messages)  latest in thread

Message-ID: <422497C8.4040109@logi-track.com>
Permalink:  ../422497C8.4040109@logi-track.com/
Also on:    postgresql.org/message-id/422497C8.4040109@logi-track.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: schabios@logi-track.com, john@arbash-meinel.com, bastiaans@sara.nl
  Subject: Re: multi billion row tables: possible or insane?
  In-Reply-To: <422497C8.4040109@logi-track.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox