pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Thor Tall <tall_thor@yahoo.com>
To: pgsql-admin@postgresql.org
Subject: Slow delete/insert.
Date: Wed, 23 Aug 2006 07:30:17 -0700 (PDT)
Message-ID: <20060823143017.98255.qmail@web50301.mail.yahoo.com> (raw)

I am in the process of converting an old system to a
new system where I have chosen to use postgres in
stead of a home grow system based on b-trees. 

The system receives 2650 message a total of 10Mbytes
of data per 15 minutes this information have to be
store in 4 tables in the database. Some of the
messages are updates to existing data and the result
is:
1038 records in table 1 size per record 282 bytes
2074 records in table 2 size per record 36 bytes
21488 records in table 3 size per record 60 bytes
25676 records in table 4 size per record 42 bytes.

Without indexes this should be something like
2,7Mbytes/15min.

The disk speed is as follows taken from hdparm -Tt
/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.89 seconds
=143.82 MB/sec
 Timing buffered disk reads:  64 MB in  1.45 seconds =
44.14 MB/sec

My problem is that the postmaster process is using
between 70-90% of the CPU time. (seen using top) and
the total wall time with nothing else running.

It takes 6 minutes to ingest the data with an empty
database and 25 minutes (wall time) if all the data is
already in the database.
The processing is done as follows:
1. Start transaction

2. check if message is in table 1 and if so delete
records from table 1(1 row),2(2 rows),3(30 rows),4(50
rows).

3. split the message and insert the date in table 1(1
row),2(2 rows),3(30 rows),4(50 rows)

4. end transaction

I need some suggestions so that I can ingest the data
faster. with the same computer and disks. We are
running redhat 7.2 on the server and I am using the c
interface to write the data to the server.

Thanks for any help,
Thor.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 



view thread (7+ messages)  latest in thread

Message-ID: <20060823143017.98255.qmail@web50301.mail.yahoo.com>
Permalink:  ../20060823143017.98255.qmail@web50301.mail.yahoo.com/
Also on:    postgresql.org/message-id/20060823143017.98255.qmail@web50301.mail.yahoo.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: tall_thor@yahoo.com
  Subject: Re: Slow delete/insert.
  In-Reply-To: <20060823143017.98255.qmail@web50301.mail.yahoo.com>
  References: <20060823143017.98255.qmail@web50301.mail.yahoo.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox