From: Steve Crawford <scrawford@pinpointresearch.com>
To: Adam Siegel <adam@sycamorehq.com>
To: pgsql-hackers@postgresql.org
To: pgsql-performance@postgresql.org
To: pgsql-general@postgresql.org
Subject: Re: realtime data inserts
Date: Mon, 12 May 2003 08:56:02 -0700
Message-ID: <20030512155602.6DD11103C7@polaris.pinpointresearch.com> (raw)
In-Reply-To: <b9j57m$236m$1@news.hub.org>
References: <b9j57m$236m$1@news.hub.org>
Depends - we don't know enough about your needs. Some questions:
Is this constant data or just capturing a burst?
Are you feeding it through one connection or several in parallel?
Did you tune your memory configs in postgresql.conf or are they still at the
minimalized defaults?
How soon does the data need to be available for query? (Obviously there will
be up to a 1200 record delay just due to the transaction.)
What generates the timestamp? Ie. is it an insert into foo values (now(),
packetname, data) or is the app providing the timestamp?
More info about the app will help.
Cheers,
Steve
On Saturday 10 May 2003 8:25 am, Adam Siegel wrote:
> I have realtime data flowing at a rate of 500, 512 byte packets per second.
> I want to log the info in a database table with two other columns, one for
> a timestamp and one for a name of the packet. The max rate I can achieve
> is 350 inserts per second on a sun blade 2000. The inserts are grouped in
> a transaction and I commit every 1200 records. I am storing the binary
> data in a bytea. I am using the libpq conversion function. Not sure if
> that is slowing me down. But I think it is the insert not the conversion.
>
> Any thoughts on how to achive this goal?
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-performance@postgresql.org
Cc: scrawford@pinpointresearch.com, adam@sycamorehq.com, pgsql-hackers@postgresql.org, pgsql-general@postgresql.org
Subject: Re: realtime data inserts
In-Reply-To: <20030512155602.6DD11103C7@polaris.pinpointresearch.com>
References: <b9j57m$236m$1@news.hub.org> <20030512155602.6DD11103C7@polaris.pinpointresearch.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox