pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedIndex use difference betweer LIKE, LIKE ANY?
6+ messages / 5 participants
[nested] [flat]
* Index use difference betweer LIKE, LIKE ANY?
@ 2011-02-25 13:31 Sam Wong <sam@hellosam.net>
0 siblings, 1 reply; 6+ messages in thread
From: Sam Wong @ 2011-02-25 13:31 UTC (permalink / raw)
To: pgsql-performance
I found that "LIKE", "= ANY (...)", "LIKE .. OR LIKE .." against a text
field used the index correctly, but not "LIKE ANY (...)". Would that be a
bug?
----
Here is my table and index:
CREATE TABLE shipment_lookup
(
shipment_id text NOT NULL,
lookup text NOT NULL
);
CREATE INDEX shipment_lookup_prefix
ONshipment_lookup
USING btree
(upper(lookup));
----
The table have 10 million rows.
The following statements use the index as expected:
select * from shipment_lookup where (UPPER(lookup) = 'SD1102228482' or
UPPER(lookup) ='ABCDEFGHIJK')
select * from shipment_lookup where (UPPER(lookup) = ANY
(ARRAY['SD1102228482','ABCDEFGHIJK']))
select * from shipment_lookup where (UPPER(lookup) LIKE 'SD1102228482%' or
UPPER(lookup) LIKE 'ABCDEFGHIJK%')
The following statement results in a full table scan (but this is what I
really want to do):
select * from shipment_lookup where (UPPER(lookup) LIKE
ANY(ARRAY['SD1102228482%', 'ABCDEFGHIJK%']))
I could rewrite the LIKE ANY(ARRAY[...]) as an LIKE .. OR .. LIKE .., but I
wonder what makes the difference?
Thanks,
Sam
----
Version Info:
Postgresql: "PostgreSQL 8.4.5, compiled by Visual C++ build 1400, 32-bit" on
Windows 2003
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Index use difference betweer LIKE, LIKE ANY?
@ 2011-02-26 21:13 Josh Berkus <josh@agliodbs.com>
parent: Sam Wong <sam@hellosam.net>
0 siblings, 1 reply; 6+ messages in thread
From: Josh Berkus @ 2011-02-26 21:13 UTC (permalink / raw)
To: Sam Wong <sam@hellosam.net>; +Cc: pgsql-performance
On 2/25/11 5:31 AM, Sam Wong wrote:
> I found that "LIKE", "= ANY (...)", "LIKE .. OR LIKE .." against a text
> field used the index correctly, but not "LIKE ANY (...)". Would that be a
> bug?
No, it would be a TODO. This is a known limitation; it needs some
clever code to make it work, and nobody's written it.
--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Index use difference betweer LIKE, LIKE ANY?
@ 2011-03-15 12:30 Chetan Suttraway <chetan.suttraway@enterprisedb.com>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 2 replies; 6+ messages in thread
From: Chetan Suttraway @ 2011-03-15 12:30 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Sam Wong <sam@hellosam.net>; pgsql-performance
On Sun, Feb 27, 2011 at 2:43 AM, Josh Berkus <josh@agliodbs.com> wrote:
> On 2/25/11 5:31 AM, Sam Wong wrote:
> > I found that "LIKE", "= ANY (...)", "LIKE .. OR LIKE .." against a text
> > field used the index correctly, but not "LIKE ANY (...)". Would that be a
> > bug?
>
> No, it would be a TODO. This is a known limitation; it needs some
> clever code to make it work, and nobody's written it.
>
>
came up with attached patch without thinking too much.
With this patch, the explain output for the same query is as below:
postgres=# explain select * from shipment_lookup where (UPPER(lookup)
LIKE
ANY(ARRAY['SD1102228482%', 'ABCDEFGHIJK%']))
;e
QUERY
PLAN
-------------------------------------------------------------------------------------------------
Seq Scan on shipment_lookup (cost=0.00..254057.36 rows=2000 width=14)
* Filter: ((upper(lookup) ~~ 'SD1102228482%'::text) OR (upper(lookup) ~~
'ABCDEFGHIJK%'::text))*
(2 rows)
postgres-#
The thing to be noted here is that the where clause "<pred> LIKE ANY
ARRAY[..]"
has been converted into
(<pred> LIKE first_array_element) or (<pred> LIKE second_array_element) or
....
Please pass on your inputs.
Regards,
Chetan
--
Chetan Sutrave
Senior Software Engineer
EnterpriseDB Corporation
The Enterprise PostgreSQL Company
Phone: +91.20.30589523
Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb
This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are not
the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.
Attachments:
like_any_array.patch (text/x-diff, 1.3K) download
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 1633499..a4a7832 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -10068,7 +10068,37 @@ a_expr: c_expr { $$ = $1; }
| a_expr subquery_Op sub_type '(' a_expr ')' %prec Op
{
if ($3 == ANY_SUBLINK)
- $$ = (Node *) makeA_Expr(AEXPR_OP_ANY, $2, $1, $5, @2);
+ {
+ bool expandLike = false;
+ /* convert expr of type "a like any(array[1,2])" into
+ * "a like 1 or a like 2"
+ */
+ if (strcmp(NameListToString($2),"~~")==0)
+ {
+ ListCell *lc;
+ if (IsA($5, A_ArrayExpr))
+ {
+ Node *res = NULL;
+ A_ArrayExpr *arr_expr = (A_ArrayExpr *)$5;
+ foreach (lc, arr_expr->elements)
+ {
+ Node *tmp = NULL;
+ Node *element = (Node*)lfirst(lc);
+ expandLike = true;
+ tmp = (Node *) makeSimpleA_Expr(AEXPR_OP, "~~", $1, element, @2);
+ if (res)
+ res = (Node*) makeA_Expr(AEXPR_OR, NIL, res, tmp, @2);
+ else
+ res = tmp;
+ }
+ $$ = res;
+ }
+
+ }
+ /* also handles array with zero elements */
+ if (!expandLike)
+ $$ = (Node *) makeA_Expr(AEXPR_OP_ANY, $2, $1, $5, @2);
+ }
else
$$ = (Node *) makeA_Expr(AEXPR_OP_ALL, $2, $1, $5, @2);
}
* Re: Index use difference betweer LIKE, LIKE ANY?
@ 2011-04-18 16:14 Robert Haas <robertmhaas@gmail.com>
parent: Chetan Suttraway <chetan.suttraway@enterprisedb.com>
1 sibling, 0 replies; 6+ messages in thread
From: Robert Haas @ 2011-04-18 16:14 UTC (permalink / raw)
To: Chetan Suttraway <chetan.suttraway@enterprisedb.com>; +Cc: Josh Berkus <josh@agliodbs.com>; Sam Wong <sam@hellosam.net>; pgsql-performance
On Tue, Mar 15, 2011 at 8:30 AM, Chetan Suttraway
<chetan.suttraway@enterprisedb.com> wrote:
> On Sun, Feb 27, 2011 at 2:43 AM, Josh Berkus <josh@agliodbs.com> wrote:
>>
>> On 2/25/11 5:31 AM, Sam Wong wrote:
>> > I found that "LIKE", "= ANY (...)", "LIKE .. OR LIKE .." against a text
>> > field used the index correctly, but not "LIKE ANY (...)". Would that be
>> > a
>> > bug?
>>
>> No, it would be a TODO. This is a known limitation; it needs some
>> clever code to make it work, and nobody's written it.
>>
>
> came up with attached patch without thinking too much.
> With this patch, the explain output for the same query is as below:
>
> postgres=# explain select * from shipment_lookup where (UPPER(lookup)
> LIKE
> ANY(ARRAY['SD1102228482%', 'ABCDEFGHIJK%']))
> ;e
> QUERY
> PLAN
> -------------------------------------------------------------------------------------------------
> Seq Scan on shipment_lookup (cost=0.00..254057.36 rows=2000 width=14)
> Filter: ((upper(lookup) ~~ 'SD1102228482%'::text) OR (upper(lookup) ~~
> 'ABCDEFGHIJK%'::text))
> (2 rows)
>
> postgres-#
>
> The thing to be noted here is that the where clause "<pred> LIKE ANY
> ARRAY[..]"
> has been converted into
> (<pred> LIKE first_array_element) or (<pred> LIKE second_array_element) or
> ....
>
> Please pass on your inputs.
Please add your patch here:
https://commitfest.postgresql.org/action/commitfest_view/open
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Index use difference betweer LIKE, LIKE ANY?
@ 2011-06-06 09:43 Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
parent: Chetan Suttraway <chetan.suttraway@enterprisedb.com>
1 sibling, 1 reply; 6+ messages in thread
From: Heikki Linnakangas @ 2011-06-06 09:43 UTC (permalink / raw)
To: Chetan Suttraway <chetan.suttraway@enterprisedb.com>; +Cc: Josh Berkus <josh@agliodbs.com>; Sam Wong <sam@hellosam.net>; pgsql-performance
On 15.03.2011 14:30, Chetan Suttraway wrote:
> On Sun, Feb 27, 2011 at 2:43 AM, Josh Berkus<josh@agliodbs.com> wrote:
>
>> On 2/25/11 5:31 AM, Sam Wong wrote:
>>> I found that "LIKE", "= ANY (...)", "LIKE .. OR LIKE .." against a text
>>> field used the index correctly, but not "LIKE ANY (...)". Would that be a
>>> bug?
>>
>> No, it would be a TODO. This is a known limitation; it needs some
>> clever code to make it work, and nobody's written it.
>>
>>
> came up with attached patch without thinking too much.
> With this patch, the explain output for the same query is as below:
>
> postgres=# explain select * from shipment_lookup where (UPPER(lookup)
> LIKE
> ANY(ARRAY['SD1102228482%', 'ABCDEFGHIJK%']))
> ;e
> QUERY
> PLAN
> -------------------------------------------------------------------------------------------------
> Seq Scan on shipment_lookup (cost=0.00..254057.36 rows=2000 width=14)
> * Filter: ((upper(lookup) ~~ 'SD1102228482%'::text) OR (upper(lookup) ~~
> 'ABCDEFGHIJK%'::text))*
> (2 rows)
>
> postgres-#
>
> The thing to be noted here is that the where clause "<pred> LIKE ANY
> ARRAY[..]"
> has been converted into
> (<pred> LIKE first_array_element) or (<pred> LIKE second_array_element) or
> ....
>
> Please pass on your inputs.
This suffers from the same multiple-evaluation issue that was recently
discovered in BETWEEN and IN expressions
(http://archives.postgresql.org/message-id/4D95B605.2020709@enterprisedb.com).
This transformation would also need to be done in the planner, after
checking that the left-hand expression is not volatile.
Also, even when safe, it's not clear that the transformation is always a
win. The left-hand expression could be expensive, in which case having
to evaluate it multiple times could hurt performance. Maybe yo
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Index use difference betweer LIKE, LIKE ANY?
@ 2011-06-06 09:47 Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
parent: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
0 siblings, 0 replies; 6+ messages in thread
From: Heikki Linnakangas @ 2011-06-06 09:47 UTC (permalink / raw)
To: Chetan Suttraway <chetan.suttraway@enterprisedb.com>; +Cc: Josh Berkus <josh@agliodbs.com>; Sam Wong <sam@hellosam.net>; pgsql-performance
On 06.06.2011 12:43, Heikki Linnakangas wrote:
> Also, even when safe, it's not clear that the transformation is always a
> win. The left-hand expression could be expensive, in which case having
> to evaluate it multiple times could hurt performance. Maybe yo
Sorry, hit "send" too early.
Maybe you could put in some heuristic to only do the transformation when
the left-hand expression is cheap, or maybe use something like the
CaseTestExpr to avoid multiple evaluation and still use the OR form.
Also, if the array is very large, opening it into the OR form could
increase plan time substantially, so we'd probably only want to do it if
there's any Vars involved, and thus any chance of matching an index.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2011-06-06 09:47 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-02-25 13:31 Index use difference betweer LIKE, LIKE ANY? Sam Wong <sam@hellosam.net>
2011-02-26 21:13 ` Josh Berkus <josh@agliodbs.com>
2011-03-15 12:30 ` Chetan Suttraway <chetan.suttraway@enterprisedb.com>
2011-04-18 16:14 ` Robert Haas <robertmhaas@gmail.com>
2011-06-06 09:43 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
2011-06-06 09:47 ` Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox