From: Bob Lunney <bob_lunney@yahoo.com>
To: pgsql-performance@postgresql.org, Jori Jovanovich <jori@dimensiology.com>
Subject: Re: SELECT ignoring index even though ORDER BY and LIMIT present
Date: Wed, 2 Jun 2010 18:49:08 -0700 (PDT)
Message-ID: <558254.61922.qm@web39704.mail.mud.yahoo.com> (raw)
In-Reply-To: <AANLkTik6RIYAtAISj1UxmzBC79l-juOuBGYSDzSkV7xG@mail.gmail.com>
Jori,
What is the PostgreSQL version/shared_buffers/work_mem/effective_cache_size/default_statistics_target? Are the statistics for the table up to date? (Run analyze verbose <tablename> to update them.) Table and index structure would be nice to know, too.
If all else fails you can set enable_seqscan = off for the session, but that is a Big Hammer for what is probably a smaller problem.
Bob Lunney
--- On Wed, 6/2/10, Jori Jovanovich <jori@dimensiology.com> wrote:
From: Jori Jovanovich <jori@dimensiology.com>
Subject: [PERFORM] SELECT ignoring index even though ORDER BY and LIMIT present
To: pgsql-performance@postgresql.org
Date: Wednesday, June 2, 2010, 4:28 PM
hi,
I have a problem space where the main goal is to search backward in time for events. Time can go back very far into the past, and so the
table can get quite large. However, the vast majority of queries are all satisfied by relatively recent data. I have an index on the row creation date and I would like almost all of my queries to have a query plan looking something like:
Limit ... -> Index Scan Backward using server_timestamp_idx on events (cost=0.00..623055.91 rows=8695 width=177)
...
However, PostgreSQL frequently tries to do a full table scan. Often what controls whether a scan is performed or not is dependent on the size of the LIMIT and how detailed the WHERE clause is. In practice, the scan is always the wrong answer for my use cases (where "always" is defined to be >99.9%).
Some examples:
(1) A sample query that devolves to a full table scan
EXPLAIN
SELECT events.id, events.client_duration, events.message, events.created_by, events.source, events.type, events.event, events.environment,
events.server_timestamp, events.session_id, events.reference, events.client_uuid FROM events
WHERE client_uuid ~* E'^foo bar so what' ORDER BY server_timestamp DESC
LIMIT 20; QUERY PLAN (BAD!)
-------------------------------------------------------------------------- Limit (cost=363278.56..363278.61 rows=20 width=177)
-> Sort (cost=363278.56..363278.62 rows=24 width=177) Sort Key: server_timestamp
-> Seq Scan on events (cost=0.00..363278.01 rows=24 width=177)
Filter: (client_uuid ~* '^foo bar so what'::text)
(2) Making the query faster by making the string match LESS specific (odd, seems like it should be MORE)
EXPLAIN
SELECT events.id, events.client_duration, events.message, events.created_by, events.source, events.type, events.event, events.environment,
events.server_timestamp, events.session_id, events.reference, events.client_uuid FROM events
WHERE client_uuid ~* E'^foo' ORDER BY server_timestamp DESC
LIMIT 20; QUERY PLAN (GOOD!)
------------------------------------------------------------------------------------------------------------ Limit (cost=0.00..1433.14 rows=20 width=177) -> Index Scan Backward using server_timestamp_idx on events (cost=0.00..623055.91 rows=8695 width=177)
Filter: (client_uuid ~* '^foo'::text)
(3) Alternatively making the query faster by using a smaller limit
EXPLAIN
SELECT events.id, events.client_duration, events.message, events.created_by, events.source, events.type, events.event, events.environment,
events.server_timestamp, events.session_id, events.reference, events.client_uuid FROM events
WHERE client_uuid ~* E'^foo bar so what' ORDER BY server_timestamp DESC
LIMIT 10;
QUERY PLAN (GOOD!) ----------------------------------------------------------------------------------------------------------
Limit (cost=0.00..259606.63 rows=10 width=177) -> Index Scan Backward using server_timestamp_idx on events (cost=0.00..623055.91 rows=24 width=177)
Filter: (client_uuid ~* '^foo bar so what'::text)
I find myself wishing I could just put a SQL HINT on the query to force the index to be used but I understand that HINTs are considered harmful and are therefore not provided for PostgreSQL, so what is the recommended way to solve this?
thank you very much
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: bob_lunney@yahoo.com, jori@dimensiology.com
Subject: Re: SELECT ignoring index even though ORDER BY and LIMIT present
In-Reply-To: <558254.61922.qm@web39704.mail.mud.yahoo.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox