X-Original-To: pgsql-general@postgresql.org Received: from localhost (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with ESMTP id 9EBFE476BCA for ; Fri, 16 May 2003 16:33:52 -0400 (EDT) Received: from postgresql.org ([64.49.215.8]) by localhost (postgresql.org [64.49.215.8:10024]) (amavisd-new) with ESMTP id 55913-03 for ; Fri, 16 May 2003 16:33:47 -0400 (EDT) Received: from pop.e-it.com (pop.e-it.com [216.187.113.81]) by postgresql.org (Postfix) with SMTP id 66581476967 for ; Fri, 16 May 2003 16:33:47 -0400 (EDT) Received: (qmail 2578 invoked from network); 16 May 2003 20:33:47 -0000 Received: from unknown (HELO ?151.191.181.108?) (216.187.113.13) by 0 with SMTP; 16 May 2003 20:33:47 -0000 Subject: Re: realtime data inserts From: Ericson Smith To: "alex b." Cc: Postgresql General In-Reply-To: <3EC549BE.5040403@gmx.de> References: <6754.1052582402@sss.pgh.pa.us> <1052591490.26574.161.camel@haggis> <10704.1052621185@sss.pgh.pa.us> <1052672090.995.13.camel@haggis> <3EC549BE.5040403@gmx.de> Content-Type: text/plain Organization: Message-Id: <1053117234.15138.23.camel@localhost.localdomain> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.2 Date: 16 May 2003 16:33:55 -0400 Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new X-Archive-Number: 200305/628 X-Sequence-Number: 42283 You probably want to have a process that constantly stores the data in a text file. Every "n" minutes, you will cause the logger to rotate the text file, then process that batch. Over here, we are able to dump around 5,000 records per second in one of our tables using that methodology. - Ericson Smith eric@did-it.com On Fri, 2003-05-16 at 16:27, alex b. wrote: > Doug McNaught wrote: > > Ron Johnson writes: > > > > > >>On Sat, 2003-05-10 at 21:46, Tom Lane wrote: > >> > >>>Ron Johnson writes: > >>> > >>>>On Sat, 2003-05-10 at 11:00, Tom Lane wrote: > >>>> > >>>>>Have you thought about using COPY? > >>> > >>>>Generate a temporary file, and then system("COPY /tmp/foobar ...") ? > >>> > >>>No, copy from stdin. No need for a temp file. > >> > >>But wouldn't that only work if the input stream is acceptable to > >>COPY ? > > > > > > Yes, but you could always pipe it through a script or C program to > > make it so... > > lets say I have an about 1kb/s continuus datastream comming in for many > hours and I'd like to store this data in my db using COPY table FROM stdin. > > At what time should I COMMIT or close the stream to feed the database > and COPY FROM again? > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html -- Ericson Smith