X-Original-To: pgsql-general@postgresql.org Received: from localhost (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with ESMTP id 788B5476903 for ; Fri, 16 May 2003 18:29:07 -0400 (EDT) Received: from postgresql.org ([64.49.215.8]) by localhost (postgresql.org [64.49.215.8:10024]) (amavisd-new) with ESMTP id 70216-05 for ; Fri, 16 May 2003 18:29:05 -0400 (EDT) Received: from lakemtao01.cox.net (lakemtao01.cox.net [68.1.17.244]) by postgresql.org (Postfix) with ESMTP id B9D7147689E for ; Fri, 16 May 2003 18:29:05 -0400 (EDT) Received: from [192.168.1.1] ([68.11.66.83]) by lakemtao01.cox.net (InterMail vM.5.01.04.05 201-253-122-122-105-20011231) with ESMTP id <20030516222908.UFC8337.lakemtao01.cox.net@[192.168.1.1]> for ; Fri, 16 May 2003 18:29:08 -0400 Subject: Re: realtime data inserts From: Ron Johnson To: Postgresql General In-Reply-To: <1053117234.15138.23.camel@localhost.localdomain> References: <6754.1052582402@sss.pgh.pa.us> <1052591490.26574.161.camel@haggis> <10704.1052621185@sss.pgh.pa.us> <1052672090.995.13.camel@haggis> <3EC549BE.5040403@gmx.de> <1053117234.15138.23.camel@localhost.localdomain> Content-Type: text/plain Organization: Message-Id: <1053124145.11845.17.camel@haggis> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.4 Date: 16 May 2003 17:29:05 -0500 Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new X-Archive-Number: 200305/631 X-Sequence-Number: 42286 On Fri, 2003-05-16 at 15:33, Ericson Smith wrote: > You probably want to have a process that constantly stores the data in a > text file. Every "n" minutes, you will cause the logger to rotate the > text file, then process that batch. Does the logger spawn the DB writer? > Over here, we are able to dump around 5,000 records per second in one of > our tables using that methodology. > > - Ericson Smith > eric@did-it.com > > On Fri, 2003-05-16 at 16:27, alex b. wrote: > > Doug McNaught wrote: > > > Ron Johnson writes: > > > > > > > > >>On Sat, 2003-05-10 at 21:46, Tom Lane wrote: > > >> > > >>>Ron Johnson writes: > > >>> > > >>>>On Sat, 2003-05-10 at 11:00, Tom Lane wrote: > > >>>> > > >>>>>Have you thought about using COPY? > > >>> > > >>>>Generate a temporary file, and then system("COPY /tmp/foobar ...") ? > > >>> > > >>>No, copy from stdin. No need for a temp file. > > >> > > >>But wouldn't that only work if the input stream is acceptable to > > >>COPY ? > > > > > > > > > Yes, but you could always pipe it through a script or C program to > > > make it so... > > > > lets say I have an about 1kb/s continuus datastream comming in for many > > hours and I'd like to store this data in my db using COPY table FROM stdin. > > > > At what time should I COMMIT or close the stream to feed the database > > and COPY FROM again? > > > > > > > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 5: Have you checked our extensive FAQ? > > > > http://www.postgresql.org/docs/faqs/FAQ.html -- +---------------------------------------------------------------+ | Ron Johnson, Jr. mailto:ron.l.johnson@cox.net | | Jefferson, LA USA http://members.cox.net/ron.l.johnson | | | | The purpose of the military isn't to pay your college tuition | | or give you a little extra income; it's to "kill people and | | break things". Surprisingly, not everyone understands that. | +---------------------------------------------------------------+