From: Ron Johnson <ron.l.johnson@cox.net>
To: Postgresql General <pgsql-general@postgresql.org>
Subject: Re: realtime data inserts
Date: 16 May 2003 17:29:05 -0500
Message-ID: <1053124145.11845.17.camel@haggis> (raw)
In-Reply-To: <1053117234.15138.23.camel@localhost.localdomain>
References: <b9j57m$236m$1@news.hub.org>
<6754.1052582402@sss.pgh.pa.us>
<1052591490.26574.161.camel@haggis>
<10704.1052621185@sss.pgh.pa.us>
<1052672090.995.13.camel@haggis>
<m3llxdju7w.fsf@varsoon.wireboard.com>
<3EC549BE.5040403@gmx.de>
<1053117234.15138.23.camel@localhost.localdomain>
On Fri, 2003-05-16 at 15:33, Ericson Smith wrote:
> You probably want to have a process that constantly stores the data in a
> text file. Every "n" minutes, you will cause the logger to rotate the
> text file, then process that batch.
Does the logger spawn the DB writer?
> Over here, we are able to dump around 5,000 records per second in one of
> our tables using that methodology.
>
> - Ericson Smith
> eric@did-it.com
>
> On Fri, 2003-05-16 at 16:27, alex b. wrote:
> > Doug McNaught wrote:
> > > Ron Johnson <ron.l.johnson@cox.net> writes:
> > >
> > >
> > >>On Sat, 2003-05-10 at 21:46, Tom Lane wrote:
> > >>
> > >>>Ron Johnson <ron.l.johnson@cox.net> writes:
> > >>>
> > >>>>On Sat, 2003-05-10 at 11:00, Tom Lane wrote:
> > >>>>
> > >>>>>Have you thought about using COPY?
> > >>>
> > >>>>Generate a temporary file, and then system("COPY /tmp/foobar ...") ?
> > >>>
> > >>>No, copy from stdin. No need for a temp file.
> > >>
> > >>But wouldn't that only work if the input stream is acceptable to
> > >>COPY ?
> > >
> > >
> > > Yes, but you could always pipe it through a script or C program to
> > > make it so...
> >
> > lets say I have an about 1kb/s continuus datastream comming in for many
> > hours and I'd like to store this data in my db using COPY table FROM stdin.
> >
> > At what time should I COMMIT or close the stream to feed the database
> > and COPY FROM again?
> >
> >
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/docs/faqs/FAQ.html--
+---------------------------------------------------------------+
| Ron Johnson, Jr. mailto:ron.l.johnson@cox.net |
| Jefferson, LA USA http://members.cox.net/ron.l.johnson |
| |
| The purpose of the military isn't to pay your college tuition |
| or give you a little extra income; it's to "kill people and |
| break things". Surprisingly, not everyone understands that. |
+---------------------------------------------------------------+
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-performance@postgresql.org
Cc: ron.l.johnson@cox.net, pgsql-general@postgresql.org
Subject: Re: realtime data inserts
In-Reply-To: <1053124145.11845.17.camel@haggis>
References: <b9j57m$236m$1@news.hub.org> <6754.1052582402@sss.pgh.pa.us> <1052591490.26574.161.camel@haggis> <10704.1052621185@sss.pgh.pa.us> <1052672090.995.13.camel@haggis> <m3llxdju7w.fsf@varsoon.wireboard.com> <3EC549BE.5040403@gmx.de> <1053117234.15138.23.camel@localhost.localdomain> <1053124145.11845.17.camel@haggis>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox