pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Joshua Marsh <icub3d@gmail.com>
To: pgsql-performance@postgresql.org
Subject: Large Database Performance suggestions
Date: Thu, 21 Oct 2004 21:14:59 -0600
Message-ID: <38242de904102120143c55ba34@mail.gmail.com> (raw)

Hello everyone,

I am currently working on a data project that uses PostgreSQL
extensively to store, manage and maintain the data.  We haven't had
any problems regarding database size until recently.  The three major
tables we use never get bigger than 10 million records.  With this
size, we can do things like storing the indexes or even the tables in
memory to allow faster access.

Recently, we have found customers who are wanting to use our service
with data files between 100 million and 300 million records.  At that
size, each of the three major tables will hold between 150 million and
700 million records.  At this size, I can't expect it to run queries
in 10-15 seconds (what we can do with 10 million records), but would
prefer to keep them all under a minute.

We did some original testing and with a server with 8GB or RAM and
found we can do operations on data file up to 50 million fairly well,
but performance drop dramatically after that.  Does anyone have any
suggestions on a good way to improve performance for these extra large
tables?  Things that have come to mind are Replication and Beowulf
clusters, but from what I have recently studied, these don't do so wel
with singular processes.  We will have parallel process running, but
it's more important that the speed of each process be faster than
several parallel processes at once.

Any help would be greatly appreciated! 

Thanks,

Joshua Marsh

P.S. Off-topic, I have a few invitations to gmail.  If anyone would
like one, let me know.



view thread (8+ messages)  latest in thread

Message-ID: <38242de904102120143c55ba34@mail.gmail.com>
Permalink:  ../38242de904102120143c55ba34@mail.gmail.com/
Also on:    postgresql.org/message-id/38242de904102120143c55ba34@mail.gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: icub3d@gmail.com
  Subject: Re: Large Database Performance suggestions
  In-Reply-To: <38242de904102120143c55ba34@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox