From: Rudi Starcevic <tech@wildcash.com>
To: Postgresql Performance list <pgsql-performance@postgresql.org>
Subject: Re: optimized counting of web statistics
Date: Wed, 29 Jun 2005 10:17:41 -0700
Message-ID: <42C2D7B5.4070707@wildcash.com> (raw)
In-Reply-To: <f3c0b408050628145545699c4e@mail.gmail.com>
References: <6EE64EF3AB31D5448D0007DD34EEB3415C2C08@Herge.rcsinc.local>
<22455e7050628133978b17cd4@mail.gmail.com>
<f3c0b408050628145545699c4e@mail.gmail.com>
Hi,
>I do my batch processing daily using a python script I've written. I
>found that trying to do it with pl/pgsql took more than 24 hours to
>process 24 hours worth of logs. I then used C# and in memory hash
>tables to drop the time to 2 hours, but I couldn't get mono installed
>on some of my older servers. Python proved the fastest and I can
>process 24 hours worth of logs in about 15 minutes. Common reports run
>in < 1 sec and custom reports run in < 15 seconds (usually).
>
>
When you say you do your batch processing in a Python script do you mean
a you are using 'plpython' inside
PostgreSQL or using Python to execut select statements and crunch the
data 'outside' PostgreSQL?
Your reply is very interesting.
Thanks.
Regards,
Rudi.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tech@wildcash.com
Subject: Re: optimized counting of web statistics
In-Reply-To: <42C2D7B5.4070707@wildcash.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox