Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 2A161632979 for ; Wed, 2 Jun 2010 22:49:23 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 59214-08 for ; Thu, 3 Jun 2010 01:49:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from web39704.mail.mud.yahoo.com (web39704.mail.mud.yahoo.com [209.191.106.50]) by mail.postgresql.org (Postfix) with SMTP id AEFD66328DF for ; Wed, 2 Jun 2010 22:49:10 -0300 (ADT) Received: (qmail 63441 invoked by uid 60001); 3 Jun 2010 01:49:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1275529748; bh=MOwBlX+AFgYcPNKMW7VvIHN1RnUP8Aq0kVH9qMu4yDc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=U3SGAXtHbzeniEj5kx+SkrnJO8XsU/1P8oqLJ23OAjcNIsxT0a3fyuC/2V3x14ZTm/lxs/YKmy1xhLfRD2tVogQZwC7+siSi56WC0ufEbwn4cDnAGcevznH9JFJN65sFODq32nY/BZsmdwMtMpPXU67uoVQbvW6DAHzcjmxQQa0= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=5Dl1rkQ9XOqACJTqBcdGlpAX9BjkxzUbj0pGqxqUXhCTk2uKJHF7z97gIQ/tFzxAz4zi5ugcs3b9uA+pAXDBUOyE6+A074QPBrZvGdAJNkWZupuHsWjSAJ45FxTETKZPx4oP4aKT/QohJViS9A2I9X1oTWahHY600yiazSAxHfA=; Message-ID: <558254.61922.qm@web39704.mail.mud.yahoo.com> X-YMail-OSG: TEilw3sVM1lL1sBHLYUZUcsuzo8.YjaU2KgMFI_sVrkZwyn q9gwqBPlW6OWjlFH2awDFhbXYx.XTCIq47I4BIZy_l.n1_V69NAeBm9wPMq4 cVPPWNqV0TwyoLi4_VesjMUAGTkwdQjd5gaLGdXq_8i7Qac1FxE8lqxtHTkD n2xVAmXkRoSXYifZ1VV8lIbKpT855Wl8OMjgBy0VQL1k57jWYYS1s32snGLg d2wkHNggRUCCOQnL3Q.sPe_VfptQ3c61jXNM3miBj9IIE6KHyLRtg86A.DUh SokfZQJPewafoR_I6eIpL0Ka1FxtRSyd0SPaRmfnB1lertJsvx4OJdW8PPm_ LVZK4vKeF1woKp2.0yJvMqcB78izBR2UeLg-- Received: from [69.76.130.83] by web39704.mail.mud.yahoo.com via HTTP; Wed, 02 Jun 2010 18:49:08 PDT X-Mailer: YahooMailClassic/11.0.8 YahooMailWebService/0.8.103.269680 Date: Wed, 2 Jun 2010 18:49:08 -0700 (PDT) From: Bob Lunney Subject: Re: SELECT ignoring index even though ORDER BY and LIMIT present To: pgsql-performance@postgresql.org, Jori Jovanovich In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-337867952-1275529748=:61922" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=1.772 tagged_above=-10 required=5 tests=BAYES_50=0.8, HTML_FONT_FACE_BAD=0.981, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: * X-Archive-Number: 201006/50 X-Sequence-Number: 38906 --0-337867952-1275529748=:61922 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Jori, What is the PostgreSQL version/shared_buffers/work_mem/effective_cache_size= /default_statistics_target?=A0 Are the statistics for the table up to date?= =A0 (Run analyze verbose to update them.)=A0 Table and index st= ructure would be nice to know, too. If all else fails you can set enable_seqscan =3D off for the session, but t= hat is a Big Hammer for what is probably a smaller problem. Bob Lunney --- On Wed, 6/2/10, Jori Jovanovich wrote: From: Jori Jovanovich Subject: [PERFORM] SELECT ignoring index even though ORDER BY and LIMIT pre= sent To: pgsql-performance@postgresql.org Date: Wednesday, June 2, 2010, 4:28 PM hi, =0AI have a problem space where the main goal is to search backward in time= for events. =A0Time can go back very far into the past, and so the=0Atable= can get quite large. =A0However, the vast majority of queries are all sati= sfied by relatively recent data. =A0I have an index on the row creation dat= e and I would like almost all of my queries to have a query plan looking so= mething like:=0A =A0Limit ...=A0=A0 -> =A0Index Scan Backward using server_timestamp_idx on = events =A0(cost=3D0.00..623055.91 rows=3D8695 width=3D177)=0A=A0=A0 =A0 =A0= =A0 ... However, PostgreSQL frequently tries to do a full table scan. =A0Often what= controls whether a scan is performed or not is dependent on the size of th= e LIMIT and how detailed the WHERE clause is. =A0In practice, the scan is a= lways the wrong answer for my use cases (where "always" is defined to be >9= 9.9%).=0A Some examples: =0A(1) A sample query that devolves to a full table scan =0A=A0=A0EXPLAIN =A0=A0 SELECT events.id, events.client_duration, events.message, events.cre= ated_by, events.source, events.type, events.event, events.environment,=0A= =A0=A0 =A0 =A0 =A0 =A0events.server_timestamp, events.session_id, events.re= ference, events.client_uuid=A0=A0 =A0 FROM events=0A=A0=A0 =A0WHERE client_= uuid ~* E'^foo bar so what'=A0ORDER BY server_timestamp DESC=0A=A0=A0 =A0LI= MIT 20;=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0QU= ERY PLAN=A0(BAD!)=0A-------------------------------------------------------= -------------------=A0Limit =A0(cost=3D363278.56..363278.61 rows=3D20 width= =3D177)=0A=A0=A0 -> =A0Sort =A0(cost=3D363278.56..363278.62 rows=3D24 width= =3D177)=A0=A0 =A0 =A0 =A0 Sort Key: server_timestamp=0A=A0=A0 =A0 =A0 =A0 -= > =A0Seq Scan on events =A0(cost=3D0.00..363278.01 rows=3D24 width=3D177)= =0A=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 Filter: (client_uuid ~* '^foo bar so what= '::text) =0A(2) Making the query faster by making the string match LESS specific (od= d, seems like it should be MORE)=0A =A0=A0EXPLAIN =A0=A0 SELECT events.id, events.client_duration, events.message, events.cre= ated_by, events.source, events.type, events.event, events.environment,=0A= =A0=A0 =A0 =A0 =A0 =A0events.server_timestamp, events.session_id, events.re= ference, events.client_uuid=A0=A0 =A0 FROM events=0A=A0=A0 =A0WHERE client_= uuid ~* E'^foo'=A0ORDER BY server_timestamp DESC=0A=A0=A0 =A0LIMIT 20;=A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0QUERY PLAN (= GOOD!) =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0=A0=0A--------------------------------------------------------------= ----------------------------------------------=A0Limit =A0(cost=3D0.00..143= 3.14 rows=3D20 width=3D177)=A0=A0 -> =A0Index Scan Backward using server_ti= mestamp_idx on events =A0(cost=3D0.00..623055.91 rows=3D8695 width=3D177)= =0A=A0=A0 =A0 =A0 =A0 Filter: (client_uuid ~* '^foo'::text) (3) Alternatively making the query faster by using a smaller limit=0A =A0=A0EXPLAIN =0A=A0=A0 SELECT events.id, events.client_duration, events.message, events.= created_by, events.source, events.type, events.event, events.environment,= =0A=A0=A0 =A0 =A0 =A0 =A0events.server_timestamp, events.session_id, events= .reference, events.client_uuid=A0=A0 =A0 FROM events=0A=A0=A0 =A0WHERE=A0cl= ient_uuid ~* E'^foo bar so what'=A0ORDER BY server_timestamp DESC=0A=A0=A0 = =A0LIMIT 10;=0A=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0QUERY PLAN=A0(GOOD!)=A0=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0----------------------------------------= ------------------------------------------------------------------=0A=A0Lim= it =A0(cost=3D0.00..259606.63 rows=3D10 width=3D177)=A0=A0 -> =A0Index Scan= Backward using server_timestamp_idx on events =A0(cost=3D0.00..623055.91 r= ows=3D24 width=3D177)=0A=A0=A0 =A0 =A0 =A0 Filter: (client_uuid ~* '^foo ba= r so what'::text) I find myself wishing I could just put a SQL HINT on the query to force the= index to be used but I understand that HINTs are considered harmful and ar= e therefore not provided for PostgreSQL, so what is the recommended way to = solve this?=0A thank you very much =0A=0A=0A=0A --0-337867952-1275529748=:61922 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Jori,

What is the PostgreSQL version/s= hared_buffers/work_mem/effective_cache_size/default_statistics_target? = ; Are the statistics for the table up to date?  (Run analyze verbose &= lt;tablename> to update them.)  Table and index structure would be = nice to know, too.

If all else fails you can set enable_seqscan =3D = off for the session, but that is a Big Hammer for what is probably a smalle= r problem.

Bob Lunney

--- On Wed, 6/2/10, Jori Jovanovich = <jori@dimensiology.com> wrote:

From: Jori Jovanovich <jori@dimensiology.com>
Subject: [PERFO= RM] SELECT ignoring index even though ORDER BY and LIMIT present
To: pgs= ql-performance@postgresql.org
Date: Wednesday, June 2, 2010, 4:28 PM

hi,

=0A
I have a problem s= pace where the main goal is to search backward in time for events.  Ti= me can go back very far into the past, and so the
=0A
table can get= quite large.  However, the vast majority of queries are all satisfied= by relatively recent data.  I have an index on the row creation date = and I would like almost all of my queries to have a query plan looking some= thing like:
=0A

 Limit ...
 =   ->  Index= Scan Backward using server_timestamp_idx on events  (cost=3D0.= 00..623055.91 rows=3D8695 width=3D177)
=0A
     &nb= sp;   ...

However, PostgreSQL frequently= tries to do a full table scan.  Often what controls whether a scan is= performed or not is dependent on the size of the LIMIT and how detailed th= e WHERE clause is.  In practice, the scan is always the wrong answer f= or my use cases (where "always" is defined to be >99.9%).
= =0A
=

=0A
(1) A sample query that devolves to a full = table scan

=0A
  EXPLAIN
   SELECT events.id, events.client_duration, events.message, events.= created_by, events.source, events.type, events.event, events.environment,
=0A
          events.server_timestamp,= events.session_id, events.reference, events.client_uuid
<= font class=3D"Apple-style-span" face=3D"'courier new', monospace"> &nb= sp;   FROM events
=0A
    WHERE client_uuid ~* E'^foo bar so what'
 ORDER BY server_timestamp DESC
=0A
   &= nbsp;LIMIT 20;
             &nb= sp;                  QUERY PLA= N (BAD!)
=0A
---------------= -----------------------------------------------------------
 = Limit  (cost=3D363278.56..363278.61 rows=3D20 width=3D177)=0A
   ->  Sort  (cost=3D363278.56..363278.62 rows=3D24= width=3D177)
         Sort Key: server_t= imestamp
=0A
         ->  Seq Scan on events  = (cost=3D0.00..363278.01 rows=3D24 width=3D177)
=0A   &nb= sp;           Filter: (client_uuid ~* '^foo bar so= what'::text)


=0A(2) Making the query faster by making the string match LESS specif= ic (odd, seems like it should be MORE)
=0A

=
  EXPLAIN
   SELECT events.id, events.client_d= uration, events.message, events.created_by, events.source, events.type, eve= nts.event, events.environment,
=0A
        = ;  events.server_timestamp, events.session_id, events.reference, event= s.client_uuid
     FROM events
=0A
=  &n= bsp;  WHERE client_= uuid ~* E'^foo'
 ORDER BY server_timestamp DESC=
=0A
    LIMIT 20;
   &= nbsp;                    =        QUERY PLAN (GOOD!)             &nb= sp;                     &= nbsp;   
=0A
-------------------------------------------= -----------------------------------------------------------------
 Limit  (cost=3D0.00..1433.14 rows=3D20 width=3D177)
&= nbsp;  ->  = Index Scan Backward using server_timestamp_idx on events  (cost= =3D0.00..623055.91 rows=3D8695 width=3D177)
=0A
    = ;     Filter: (client_uuid ~* '^foo'::text)

<= div>
(3) Alte= rnatively making the query faster by using a smaller limit=0A

  EXPLAIN
=0A
   SELECT events.id, event= s.client_duration, events.message, events.created_by, events.source, events= .type, events.event, events.environment,
=0A
     &n= bsp;    events.server_timestamp, events.session_id, events.refere= nce, events.client_uuid
     FROM events=0A
 ORDER BY server_timestamp DESC
=0A
    LIMIT = 10;
<= /div>
(GOOD!)  =                      = ;                
------------= ---------------------------------------------------------------------------= -------------------
=0A
 Limit  (cost=3D0.00..259606.63 = rows=3D10 width=3D177)
   ->  Index Scan Backward using server_t= imestamp_idx on events  (cost=3D0.00..623055.91 rows=3D24 width=3D177)=
=0A
         Filter: (client_uuid ~* '^f= oo bar so what'::text)


I find mysel= f wishing I could just put a SQL HINT on the query to force the index to be= used but I understand that HINTs are considered harmful and are therefore = not provided for PostgreSQL, so what is the recommended way to solve this?<= /div>=0A

thank you very much

=
=0A
=0A=0A --0-337867952-1275529748=:61922--