Message-ID: <4DD0195A.9040305@agliodbs.com>
Date: Sun, 15 May 2011 11:20:10 -0700
From: Josh Berkus <josh@agliodbs.com>
Organization: PostgreSQL Experts Inc.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
	rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Re: reducing random_page_cost from 4 to 2 to force index
 scan
References: <BANLkTikZhYc+h1=YDJfXwZEZc1tN3_VwCg@mail.gmail.com>
	<4DB6F4BB020000250003CEB9@gw.wicourts.gov>
	<BANLkTindB68ogMg8zrR7GrYEV+mQQddj0A@mail.gmail.com>
	<BANLkTik6OaVkJGEjs5--vcynUE36mWbzbg@mail.gmail.com>
	<2823.1305314437@sss.pgh.pa.us>
	<BANLkTimTqXAy7_KqQz_OW1RvMbDd9ojfqg@mail.gmail.com>
	<4DCD90F5.2010501@agliodbs.com>
	<BANLkTikcVUeRfocJPe_W+jfi5k1+gcLnxg@mail.gmail.com>
In-Reply-To: <BANLkTikcVUeRfocJPe_W+jfi5k1+gcLnxg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Stuart,

> I think random_page_cost causes problems because I need to combine
> disk random access time, which I can measure, with a guesstimate of
> the disk cache hit rate.

See, that's wrong. Disk cache hit rate is what effective_cache_size
(ECS) is for.

Really, there's several factors which should be going into the planner's
estimates to determine a probability of a table being cached:

* ratio between total database size and ECS
* ratio between table size and ECS
* ratio between index size and ECS
* whether the table is "hot" or not
* whether the index is "hot" or not

The last two statistics are critically important for good estimation,
and they are not things we currently collect.  By "hot" I mean: is this
a relation which is accessed several times per minute/hour and is thus
likely to be in the cache when we need it?  Currently, we have no way of
knowing that.

Without "hot" statistics, we're left with guessing based on size, which
results in bad plans for small tables in large databases which are
accessed infrequently.

Mind you, for large tables it would be even better to go beyond that and
actually have some knowledge of which disk pages might be in cache.
However, I think that's beyond feasibility for current software/OSes.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com