From: Andy Colson <andy@squeakycode.net>
To: Tory M Blue <tmblue@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Performance question 83 GB Table 150 million rows, distinct select
Date: Wed, 16 Nov 2011 20:11:18 -0600
Message-ID: <4EC46D46.5050608@squeakycode.net> (raw)
In-Reply-To: <CAEaSS0b7D4eyDi4kt-Dt8u-yGq+akD=5PZwMk_TUHP-zujAnCQ@mail.gmail.com>
References: <CAEaSS0b7D4eyDi4kt-Dt8u-yGq+akD=5PZwMk_TUHP-zujAnCQ@mail.gmail.com>
On 11/16/2011 04:53 PM, Tory M Blue wrote:
> Linux F12 64bit
> Postgres 8.4.4
> 16 proc / 32GB
> 8 disk 15KRPM SAS/Raid 5 (I know!)
>
>
> shared_buffers = 6000MB
> #temp_buffers = 8MB
> max_prepared_transactions = 0
> work_mem = 250MB
> maintenance_work_mem = 1000MB
>
>
>
>
>
> We now have about 180mill records in that table. The database size is
> about 580GB and the userstats table which is the biggest one and the
> one we query the most is 83GB.
>
> Just a basic query takes 4 minutes:
>
> For e.g. select count(distinct uid) from userstats where log_date>'11/7/2011'
>
How'd you feel about keeping a monthly summary table? Update it daily, with only a days worth of stats, then you could query the summary table much faster.
That's what I do for my website stats. I log details for a month, then summarize everything into a summary table, and blow away the details. You wouldn't have to delete the details if you wanted them, just keeping the summary table updated would be enough.
-Andy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: andy@squeakycode.net, tmblue@gmail.com
Subject: Re: Performance question 83 GB Table 150 million rows, distinct select
In-Reply-To: <4EC46D46.5050608@squeakycode.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox