To: Karl Denninger <karl@denninger.net>
cc: Kevin Grittner <Kevin.Grittner@wicourts.gov>,
	pgsql-performance@postgresql.org, pgsql-sql@postgresql.org
Subject: Re: [SQL] SQL Query Performance - what gives? 
In-reply-to: <4A8B334F.1010801@denninger.net> 
References: <4A8B04C7.6060702@denninger.net>
	<4A8AC75C0200002500029C8A@gw.wicourts.gov>
	<4A8B1606.2040209@denninger.net>
	<4A8ADDEA0200002500029CA3@gw.wicourts.gov>
	<4A8B334F.1010801@denninger.net>
Comments: In-reply-to Karl Denninger <karl@denninger.net>
	message dated "Tue, 18 Aug 2009 18:03:44 -0500"
Date: Tue, 18 Aug 2009 22:02:18 -0400
Message-ID: <9800.1250647338@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>

Karl Denninger <karl@denninger.net> writes:
> The problem appearsa to lie in the "nested loop", and I don't understand
> why that's happening.

It looks to me like there are several issues here.

One is the drastic underestimate of the number of rows satisfying the
permission condition. That leads the planner to think that a nestloop
join with the other table will be fast, which is only right if there are
just one or a few rows coming out of "forum".  With sixty-some rows you
get sixty-some repetitions of the scan of the other table, which loses.

Problem number two is the overeager use of a BitmapAnd to add on another
index that isn't really very selective.  That might be a correct
decision but it looks fishy here.  We rewrote choose_bitmap_and a couple
of times to try to fix that problem ... what PG version is this exactly?

The third thing that looks fishy is that it's using unqualified index
scans for no apparent reason.  Have you got enable_seqscan turned off,
and if so what happens when you fix that?  What other nondefault planner
settings are you using?

But anyway, the big problem seems to be poor selectivity estimates for 
conditions like "(permission & 127) = permission".  I have bad news for
you: there is simply no way in the world that Postgres is not going to
suck at estimating that, because the planner has no knowledge whatsoever
of the behavior of "&".  You could consider writing and submitting a
patch that would teach it something about that, but in the near term
it would be a lot easier to reconsider your representation of
permissions.  You'd be likely to get significantly better results,
not to mention have more-readable queries, if you stored them as a group
of simple boolean columns.

			regards, tom lane