From: Ericson Smith <eric@did-it.com>
To: alex b. <mailinglists1@gmx.de>
Cc: Postgresql General <pgsql-general@postgresql.org>
Subject: Re: realtime data inserts
Date: 16 May 2003 16:33:55 -0400
Message-ID: <1053117234.15138.23.camel@localhost.localdomain> (raw)
In-Reply-To: <3EC549BE.5040403@gmx.de>
References: <b9j57m$236m$1@news.hub.org>
<6754.1052582402@sss.pgh.pa.us>
<1052591490.26574.161.camel@haggis>
<10704.1052621185@sss.pgh.pa.us>
<1052672090.995.13.camel@haggis>
<m3llxdju7w.fsf@varsoon.wireboard.com>
<3EC549BE.5040403@gmx.de>
You probably want to have a process that constantly stores the data in a
text file. Every "n" minutes, you will cause the logger to rotate the
text file, then process that batch.
Over here, we are able to dump around 5,000 records per second in one of
our tables using that methodology.
- Ericson Smith
eric@did-it.com
On Fri, 2003-05-16 at 16:27, alex b. wrote:
> Doug McNaught wrote:
> > Ron Johnson <ron.l.johnson@cox.net> writes:
> >
> >
> >>On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
> >>
> >>>Ron Johnson <ron.l.johnson@cox.net> writes:
> >>>
> >>>>On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
> >>>>
> >>>>>Have you thought about using COPY?
> >>>
> >>>>Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
> >>>
> >>>No, copy from stdin. No need for a temp file.
> >>
> >>But wouldn't that only work if the input stream is acceptable to
> >>COPY ?
> >
> >
> > Yes, but you could always pipe it through a script or C program to
> > make it so...
>
> lets say I have an about 1kb/s continuus datastream comming in for many
> hours and I'd like to store this data in my db using COPY table FROM stdin.
>
> At what time should I COMMIT or close the stream to feed the database
> and COPY FROM again?
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html--
Ericson Smith <eric@did-it.com>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-performance@postgresql.org
Cc: eric@did-it.com, mailinglists1@gmx.de, pgsql-general@postgresql.org
Subject: Re: realtime data inserts
In-Reply-To: <1053117234.15138.23.camel@localhost.localdomain>
References: <b9j57m$236m$1@news.hub.org> <6754.1052582402@sss.pgh.pa.us> <1052591490.26574.161.camel@haggis> <10704.1052621185@sss.pgh.pa.us> <1052672090.995.13.camel@haggis> <m3llxdju7w.fsf@varsoon.wireboard.com> <3EC549BE.5040403@gmx.de> <1053117234.15138.23.camel@localhost.localdomain>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox