pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedLogarithmic data frequency distributions and the query planner
2+ messages / 2 participants
[nested] [flat]
* Logarithmic data frequency distributions and the query planner
@ 2010-07-07 20:54 Jerry Gamache <jerry.gamache@idilia.com>
2010-07-07 21:22 ` Re: Logarithmic data frequency distributions and the query planner Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 2+ messages in thread
From: Jerry Gamache @ 2010-07-07 20:54 UTC (permalink / raw)
To: pgsql-performance
On 8.1, I have a very interesting database where the distributions of
some values in a multi-million rows table is logarithmic (i.e. the most
frequent value is an order of magnitude more frequent than the next
ones). If I analyze the table, the statistics become extremely skewed
towards the most frequent values and this prevents the planner from
giving any good results on queries that do not target these entries.
In a recent case, the planner estimated that the number of returned rows
would be ~13% of the table size and from this bad assumption generated a
very slow conservative plan that executed in days. If I set the
statistics at zero for that table, the planner uses a hardcoded ratio
(seems like 0.5%) for the number of returned rows and this helps
generating a plan that executes in 3 minutes (still sub-optimal, but not
as bad).
Generating partial index for the less frequent cases helps, but this
solution is not flexible enough for our needs as the number of complex
queries grow. We are mostly left with pre-generating a lot of temporary
tables whenever the planner over-evaluates the number of rows generated
by a subquery (query execution was trimmed from 3 minutes to 30 seconds
using this technique) or using the OFFSET 0 tweak, but it would be nice
if the planner could handle this on its own.
Am I missing something obvious? Setting the statistics for this table to
zero seems awkward even if it gives good results.
Jerry.
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Logarithmic data frequency distributions and the query planner
2010-07-07 20:54 Logarithmic data frequency distributions and the query planner Jerry Gamache <jerry.gamache@idilia.com>
@ 2010-07-07 21:22 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 2+ messages in thread
From: Tom Lane @ 2010-07-07 21:22 UTC (permalink / raw)
To: Jerry Gamache <jerry.gamache@idilia.com>; +Cc: pgsql-performance
Jerry Gamache <jerry.gamache@idilia.com> writes:
> On 8.1, I have a very interesting database where the distributions of
> some values in a multi-million rows table is logarithmic (i.e. the most
> frequent value is an order of magnitude more frequent than the next
> ones). If I analyze the table, the statistics become extremely skewed
> towards the most frequent values and this prevents the planner from
> giving any good results on queries that do not target these entries.
Highly skewed distributions are hardly unusual, and I'm not aware that
the planner is totally incapable of dealing with them. You do need a
large enough stats target to get down into the tail of the
distribution (the default target for 8.1 is probably too small for you).
It might be that there have been some other relevant improvements since
8.1, too ...
regards, tom lane
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2010-07-07 21:22 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-07-07 20:54 Logarithmic data frequency distributions and the query planner Jerry Gamache <jerry.gamache@idilia.com>
2010-07-07 21:22 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox