From: Luke Lonergan <llonergan@greenplum.com>
To: Ulrich Wisser <ulrich.wisser@relevanttraffic.se>
To: pgsql-performance@postgresql.org
Cc: Nicholas E. Wakefield <nwakefield@KineticNetworks.com>
Cc: Barry Klawans <bklawans@jaspersoft.com>
Cc: Daria Hutchinson <dhutchinson@greenplum.com>
Subject: Re: Need for speed 3
Date: Thu, 01 Sep 2005 09:37:53 -0700
Message-ID: <BF3C7C71.EBC5%llonergan@greenplum.com> (raw)
In-Reply-To: <43170151.90408@relevanttraffic.se>
Ulrich,
On 9/1/05 6:25 AM, "Ulrich Wisser" <ulrich.wisser@relevanttraffic.se> wrote:
> My application basically imports Apache log files into a Postgres
> database. Every row in the log file gets imported in one of three (raw
> data) tables. My columns are exactly as in the log file. The import is
> run approx. every five minutes. We import about two million rows a month.
Bizgres Clickstream does this job using an ETL (extract transform and load)
process to transform the weblogs into an optimized schema for reporting.
> After every import the data from the current day is deleted from the
> reporting table and recalculated from the raw data table.
This is something the optimized ETL in Bizgres Clickstream also does well.
> What do you think of this approach? Are there better ways to do it? Is
> there some literature you recommend reading?
I recommend the Bizgres Clickstream docs, you can get it from Bizgres CVS,
and there will shortly be a live html link on the website.
Bizgres is free - it also improves COPY performance by almost 2x, among
other enhancements.
- Luke
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: llonergan@greenplum.com, ulrich.wisser@relevanttraffic.se, nwakefield@KineticNetworks.com, bklawans@jaspersoft.com, dhutchinson@greenplum.com
Subject: Re: Need for speed 3
In-Reply-To: <BF3C7C71.EBC5%llonergan@greenplum.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox