To: "Dave Held" <dave.held@arraysg.com>
Cc: "pgsql-perform" <pgsql-performance@postgresql.org>,
	<pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Bad n_distinct estimation; hacks suggested?
References: <49E94D0CFCD4DB43AFBA928DDD20C8F9026184E1@asg002.asg.local>
In-Reply-To: <49E94D0CFCD4DB43AFBA928DDD20C8F9026184E1@asg002.asg.local>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 27 Apr 2005 13:16:48 -0400
Message-ID: <87ekcw3zsv.fsf@stark.xeocode.com>
Lines: 24
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii


"Dave Held" <dave.held@arraysg.com> writes:

> > Actually, it's more to characterize how large of a sample
> > we need.  For example, if we sample 0.005 of disk pages, and
> > get an estimate, and then sample another 0.005 of disk pages
> > and get an estimate which is not even close to the first
> > estimate, then we have an idea that this is a table which 
> > defies analysis based on small samples.  
> 
> I buy that.

Better yet is to use the entire sample you've gathered of .01 and then perform
analysis on that sample to see what the confidence interval is. Which is
effectively the same as what you're proposing except looking at every possible
partition.

Unfortunately the reality according to the papers that were sent earlier is
that you will always find the results disappointing. Until your sample is
nearly the entire table your estimates for n_distinct will be extremely
unreliable.

-- 
greg