Message-ID: <1114580284.426f253cc0087@webmail.telus.net>
Date: Tue, 26 Apr 2005 22:38:04 -0700
From: Mischa Sandberg <mischa.sandberg@telus.net>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: josh@agliodbs.com, pgsql-perform <pgsql-performance@postgresql.org>,
	pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Bad n_distinct estimation; hacks suggested?
References: <200504191209.05181.josh@agliodbs.com>
	<19276.1114442580@sss.pgh.pa.us>
	<1114454941.21529.245.camel@localhost.localdomain>
	<200504251213.18565.josh@agliodbs.com>
	<426D565E.8040400@dunslane.net>
In-Reply-To: <426D565E.8040400@dunslane.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
User-Agent: Internet Messaging Program (IMP) 3.1-cvs

Quoting Andrew Dunstan <andrew@dunslane.net>: 
 
> After some more experimentation, I'm wondering about some sort of  
> adaptive algorithm, a bit along the lines suggested by Marko 
Ristola, but limited to 2 rounds. 
>  
> The idea would be that we take a sample (either of fixed size, or 
> some  small proportion of the table) , see how well it fits a larger 
sample 
> > (say a few times the size of the first sample), and then adjust 
the > formula accordingly to project from the larger sample the 
estimate for the full population. Math not worked out yet - I think we 
want to ensure that the result remains bounded by [d,N]. 
 
Perhaps I can save you some time (yes, I have a degree in Math). If I 
understand correctly, you're trying extrapolate from the correlation 
between a tiny sample and a larger sample. Introducing the tiny sample 
into any decision can only produce a less accurate result than just 
taking the larger sample on its own; GIGO. Whether they are consistent 
with one another has no relationship to whether the larger sample 
correlates with the whole population. You can think of the tiny sample 
like "anecdotal" evidence for wonderdrugs.  
--  
"Dreams come true, not free." -- S.Sondheim, ITW