X-Original-To: pgsql-general@postgresql.org Received: from spampd.localdomain (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with ESMTP id B591747580B; Mon, 12 May 2003 11:56:02 -0400 (EDT) Received: from polaris.pinpointresearch.com (66-7-238-176.cust.telepacific.net [66.7.238.176]) by postgresql.org (Postfix) with ESMTP id A16CD474E42; Mon, 12 May 2003 11:55:57 -0400 (EDT) Received: from there (66-7-238-179.cust.telepacific.net [66.7.238.179]) by polaris.pinpointresearch.com (Postfix) with SMTP id 6DD11103C7; Mon, 12 May 2003 08:56:02 -0700 (PDT) Content-Type: text/plain; charset="iso-8859-1" From: Steve Crawford Organization: Pinpoint Research To: "Adam Siegel" , pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org, pgsql-general@postgresql.org Subject: Re: realtime data inserts Date: Mon, 12 May 2003 08:56:02 -0700 X-Mailer: KMail [version 1.3.1] References: In-Reply-To: MIME-Version: 1.0 Message-Id: <20030512155602.6DD11103C7@polaris.pinpointresearch.com> Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, hits=-32.5 required=5.0 tests=BAYES_01,EMAIL_ATTRIBUTION,IN_REP_TO,QUOTED_EMAIL_TEXT, REFERENCES,REPLY_WITH_QUOTES autolearn=ham version=2.50 X-Spam-Level: X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-Archive-Number: 200305/357 X-Sequence-Number: 42012 Depends - we don't know enough about your needs. Some questions: Is this constant data or just capturing a burst? Are you feeding it through one connection or several in parallel? Did you tune your memory configs in postgresql.conf or are they still at th= e=20 minimalized defaults? How soon does the data need to be available for query? (Obviously there wil= l=20 be up to a 1200 record delay just due to the transaction.) What generates the timestamp? Ie. is it an insert into foo values (now(),= =20 packetname, data) or is the app providing the timestamp? More info about the app will help. Cheers, Steve On Saturday 10 May 2003 8:25 am, Adam Siegel wrote: > I have realtime data flowing at a rate of 500, 512 byte packets per secon= d. > I want to log the info in a database table with two other columns, one for > a timestamp and one for a name of the packet. The max rate I can achieve > is 350 inserts per second on a sun blade 2000. The inserts are grouped in > a transaction and I commit every 1200 records. I am storing the binary > data in a bytea. I am using the libpq conversion function. Not sure if > that is slowing me down. But I think it is the insert not the conversion. > > Any thoughts on how to achive this goal? > > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster