pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Heikki Linnakangas <hlinnakangas@vmware.com>
To: Tomas Vondra <tv@fuzzy.cz>
Cc: pgsql-hackers@postgresql.org
Subject: Re: GIN improvements part2: fast scan
Date: Thu, 30 Jan 2014 17:38:18 +0100
Message-ID: <52EA7FFA.60001@vmware.com> (raw)
In-Reply-To: <52E9A274.6040806@fuzzy.cz>
References: <CAPpHfdvftaJq7www381naLw1=4u0h+qpXgWvNhcEB9HMVywbGg@mail.gmail.com>
	<CAPpHfduJ-X9WKR9h-vyTrNEVNR37ZDx05qOLkA3ys3j-14uKXA@mail.gmail.com>
	<CAPpHfduRrMOvXNrmJ0+ALk9dDR3Z34HpNdQPkQZtF_oYX58h9A@mail.gmail.com>
	<51D00FB6.9030008@vmware.com>
	<CAPpHfdtsDe-O6oQdCZUh6N1gGOMz0XKTMNQro-fueg-R06PaWw@mail.gmail.com>
	<528533D9.3040501@vmware.com>
	<CAPpHfduOvUFT+GgCrnBOzRR19eTb9mDsDJwU=S0h477fdgr=RA@mail.gmail.com>
	<CAPpHfdtZOhsmz_uf5BfdXcj5W3zk3K-PYxFc8-7eQw-nPXX1DA@mail.gmail.com>
	<CAPpHfdsEKFmrOU51EWSbF-RDESasSVa2YjoyFoG+0jCZOn1qjw@mail.gmail.com>
	<CAPpHfdvs-EHiwmTgfK=uJ4W3OjskVHyfkuehtyrKfKY_KmMpVw@mail.gmail.com>
	<52E141BA.5000302@vmware.com>
	<52E4141E.60609@fuzzy.cz>
	<52E42AD0.5060600@vmware.com>
	<52E4AA3A.9000607@fuzzy.cz>
	<52E53458.2010106@vmware.com>
	<CAPpHfdticvY2+RGTxzQdrr4YSQie+vYGRDRBTuG1ZxBmjwGrOA@mail.gmail.com>
	<CAPpHfduivz3MOA8_o+44n_Z0P-n+WCW=K1DrH+-g3gb-cbM2aA@mail.gmail.com>
	<52E72A10.5040908@fuzzy.cz>
	<52E75C54.9050209@vmware.c om>
	<52E9A274.6040806@fuzzy.c z>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>

On 01/30/2014 01:53 AM, Tomas Vondra wrote:
> (3) A file with explain plans for 4 queries suffering ~2x slowdown,
>      and explain plans with 9.4 master and Heikki's patches is available
>      here:
>
>        http://www.fuzzy.cz/tmp/gin/queries.txt
>
>      All the queries have 6 common words, and the explain plans look
>      just fine to me - exactly like the plans for other queries.
>
>      Two things now caught my eye. First some of these queries actually
>      have words repeated - either exactly like "database & database" or
>      in negated form like "!anything & anything". Second, while
>      generating the queries, I use "dumb" frequency, where only exact
>      matches count. I.e. "write != written" etc. But the actual number
>      of hits may be much higher - for example "write" matches exactly
>      just 5% documents, but using @@ it matches more than 20%.
>
>      I don't know if that's the actual cause though.

I tried these queries with the data set you posted here: 
http://www.postgresql.org/message-id/52E4141E.60609@fuzzy.cz. The reason 
for the slowdown is the extra consistent calls it causes. That's 
expected - the patch certainly does call consistent in situations where 
it didn't before, and if the "pre-consistent" checks are not able to 
eliminate many tuples, you lose.

So, what can we do to mitigate that? Some ideas:

1. Implement the catalog changes from Alexander's patch. That ought to 
remove the overhead, as you only need to call the consistent function 
once, not "both ways". OTOH, currently we only call the consistent 
function if there is only one unknown column. If with the catalog 
changes, we always call the consistent function even if there are more 
unknown columns, we might end up calling it even more often.

2. Cache the result of the consistent function.

3. Make the consistent function cheaper. (How? Magic?)

4. Use some kind of a heuristic, and stop doing the pre-consistent 
checks if they're not effective. Not sure what the heuristic would look 
like.

The caching we could easily do. It's very simple and very effective, as 
long as the number of number of entries is limited. The amount of space 
required to cache all combinations grows exponentially, so it's only 
feasible for up to 10 or so entries.

- Heikki


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers



view thread (90+ messages)  latest in thread

Message-ID: <52EA7FFA.60001@vmware.com>
Permalink:  ../52EA7FFA.60001@vmware.com/
Also on:    postgresql.org/message-id/52EA7FFA.60001@vmware.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: hlinnakangas@vmware.com, tv@fuzzy.cz
  Subject: Re: GIN improvements part2: fast scan
  In-Reply-To: <52EA7FFA.60001@vmware.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox