From: Alan Hodgson <ahodgson@simkin.ca>
Organization: Simkin Network Consulting
To: pgsql-performance@postgresql.org
Subject: Re: Performance question 83 GB Table 150 million rows,
 distinct select
Date: Wed, 16 Nov 2011 15:27:57 -0800
User-Agent: KMail/1.13.6 (Linux/2.6.38-12-generic; KDE/4.6.5; x86_64; ; )
References: 
 <CAEaSS0b7D4eyDi4kt-Dt8u-yGq+akD=5PZwMk_TUHP-zujAnCQ@mail.gmail.com>
In-Reply-To: 
 <CAEaSS0b7D4eyDi4kt-Dt8u-yGq+akD=5PZwMk_TUHP-zujAnCQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201111161527.57443.ahodgson@simkin.ca>

On November 16, 2011 02:53:17 PM Tory M Blue wrote:
> We now have about 180mill records in that table. The database size is
> about 580GB and the userstats table which is the biggest one and the
> one we query the most is 83GB.
> 
> Just a basic query takes 4 minutes:
> 
> For e.g. select count(distinct uid) from userstats where log_date
> >'11/7/2011'
>
> Just not sure if this is what to expect, however there are many other
> DB's out there bigger than ours, so I'm curious what can I do?

That query should use an index on log_date if one exists. Unless the planner 
thinks it would need to look at too much of the table.

Also, the normal approach to making large statistics tables more manageable is 
to partition them by date range.