Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 240E264FD03 for ; Sun, 10 Aug 2008 00:30:57 -0300 (ADT) Received: from postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 37572-03-3 for ; Sun, 10 Aug 2008 00:30:25 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.173]) by postgresql.org (Postfix) with ESMTP id 70F4D64FFAE for ; Sun, 10 Aug 2008 00:25:08 -0300 (ADT) Received: by wf-out-1314.google.com with SMTP id 25so1332017wfc.28 for ; Sat, 09 Aug 2008 20:25:07 -0700 (PDT) Received: by 10.143.32.7 with SMTP id k7mr1560257wfj.305.1218338707434; Sat, 09 Aug 2008 20:25:07 -0700 (PDT) Received: by 10.143.39.19 with HTTP; Sat, 9 Aug 2008 20:25:07 -0700 (PDT) Message-ID: Date: Sat, 9 Aug 2008 20:25:07 -0700 From: "Scott Carey" To: Miernik Subject: Re: why query plan for the inner SELECT of WHERE x IN is wrong, but when run the inner query alone is OK? Cc: pgsql-performance@postgresql.org In-Reply-To: <20080810022059.7330.0.NOFFLE@turbacz.local> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_74517_18069023.1218338707439" References: <20080809203435.7227.0.NOFFLE@turbacz.local> <20080809223208.7297.0.NOFFLE@turbacz.local> <20080810022059.7330.0.NOFFLE@turbacz.local> X-Google-Sender-Auth: a282fc4bde431470 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 200808/94 X-Sequence-Number: 30975 ------=_Part_74517_18069023.1218338707439 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline It is hardware dependent. The estimates are not time estimates, but on an arbitrary scale. On the server I work with, the estimates are almost always 10x larger than the run times, and sometimes more than 50x. (many GBs RAM, 8 CPU cores, more than 10 disks, standard optimizer settings other than statistics sample sizes and increased common values for columns). -Scott On Sat, Aug 9, 2008 at 7:20 PM, Miernik wrote: > Miernik wrote: > > I present a SELECT uid plan with the 1000 table also below, just to be > > sure, this is the "bad" plan, that takes forever: > > > > miernik=> EXPLAIN SELECT uid FROM cnts WHERE uid IN (SELECT uid FROM alog > WHERE pid = 3452654 AND o = 1); > > QUERY PLAN > > > ----------------------------------------------------------------------------------------------- > > Nested Loop IN Join (cost=0.00..3532.70 rows=1 width=4) > > -> Seq Scan on cnts (cost=0.00..26.26 rows=1026 width=4) > > -> Index Scan using alog_uid_idx on alog (cost=0.00..297.32 rows=1 > width=4) > > Index Cond: ((alog.uid)::integer = (cnts.uid)::integer) > > Filter: ((alog.pid = 3452654::numeric) AND (alog.o = 1::numeric)) > > (5 rows) > > If I reduce the number of rows in cnts to 100, I can actually make an > EXPLAIN ANALYZE with this query plan finish in reasonable time: > > miernik=> EXPLAIN ANALYZE SELECT uid FROM cnts WHERE uid IN (SELECT uid > FROM alog WHERE pid = 555949 AND odp = 1); > QUERY > PLAN > > ------------------------------------------------------------------------------------------------------------------------------------------------- > Nested Loop IN Join (cost=0.00..3585.54 rows=1 width=4) (actual > time=51831.430..267844.815 rows=7 loops=1) > -> Seq Scan on cnts (cost=0.00..14.00 rows=700 width=4) (actual > time=0.005..148.464 rows=100 loops=1) > -> Index Scan using alog_uid_idx on alog (cost=0.00..301.02 rows=1 > width=4) (actual time=2676.959..2676.959 rows=0 loops=100) > Index Cond: ((alog.uid)::integer = (cnts.uid)::integer) > Filter: ((alog.pid = 555949::numeric) AND (alog.odp = 1::numeric)) > Total runtime: 267844.942 ms > (6 rows) > > The real running times are about 10 times more than the estimates. Is > that normal? > > -- > Miernik > http://miernik.name/ > > > -- > Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance > ------=_Part_74517_18069023.1218338707439 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline
It is hardware dependent.   The estimates are not time estimates, but on an arbitrary scale.

On the server I work with, the estimates are almost always 10x larger than the run times, and sometimes more than 50x.

(many GBs RAM, 8 CPU cores, more than 10 disks, standard optimizer settings other than statistics sample sizes and increased common values for columns).

-Scott

On Sat, Aug 9, 2008 at 7:20 PM, Miernik <public@public.miernik.name> wrote:
Miernik <public@public.miernik.name> wrote:
> I present a SELECT uid plan with the 1000 table also below, just to be
> sure, this is the "bad" plan, that takes forever:
>
> miernik=> EXPLAIN SELECT uid FROM cnts WHERE uid IN (SELECT uid FROM alog WHERE pid = 3452654 AND o = 1);
>                                          QUERY PLAN
> -----------------------------------------------------------------------------------------------
> Nested Loop IN Join  (cost=0.00..3532.70 rows=1 width=4)
>   ->  Seq Scan on cnts  (cost=0.00..26.26 rows=1026 width=4)
>   ->  Index Scan using alog_uid_idx on alog  (cost=0.00..297.32 rows=1 width=4)
>         Index Cond: ((alog.uid)::integer = (cnts.uid)::integer)
>         Filter: ((alog.pid = 3452654::numeric) AND (alog.o = 1::numeric))
> (5 rows)

If I reduce the number of rows in cnts to 100, I can actually make an
EXPLAIN ANALYZE with this query plan finish in reasonable time:

miernik=> EXPLAIN ANALYZE SELECT uid FROM cnts WHERE uid IN (SELECT uid FROM alog WHERE pid = 555949 AND odp = 1);
                                                                  QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop IN Join  (cost=0.00..3585.54 rows=1 width=4) (actual time=51831.430..267844.815 rows=7 loops=1)
  ->  Seq Scan on cnts  (cost=0.00..14.00 rows=700 width=4) (actual time=0.005..148.464 rows=100 loops=1)
  ->  Index Scan using alog_uid_idx on alog  (cost=0.00..301.02 rows=1 width=4) (actual time=2676.959..2676.959 rows=0 loops=100)
        Index Cond: ((alog.uid)::integer = (cnts.uid)::integer)
        Filter: ((alog.pid = 555949::numeric) AND (alog.odp = 1::numeric))
 Total runtime: 267844.942 ms
(6 rows)

The real running times are about 10 times more than the estimates. Is
that normal?

--
Miernik
http://miernik.name/


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

------=_Part_74517_18069023.1218338707439--