pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feed2 simple SQL questions: optimizing aggegate query
5+ messages / 2 participants
[nested] [flat]
* 2 simple SQL questions: optimizing aggegate query
@ 2003-05-05 18:59 Alex Rice <alrice@ARCplanning.com>
2003-05-05 19:45 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
0 siblings, 1 reply; 5+ messages in thread
From: Alex Rice @ 2003-05-05 18:59 UTC (permalink / raw)
To: pgsql-sql
Hello, I am using Mnogosearch with a PostgreSQL backend. I am writing
my own queries because I can't use the existing Perl or PHP front ends
to Mnogosearch.
1) Is there a way to optimize this query? It takes ~6 seconds on my
workstation :-( I would like to get it under 2 seconds.
SELECT url.rec_id, url, title, content_type, txt,
sum(
case
when dict.word = 'wordx' then 1
when dict.word = 'wordx' then 1
when dict.word = 'wordy' then 1
else 0
end
) as rank
FROM dict, url
WHERE url.rec_id = dict.url_id
GROUP BY rec_id, url, title, content_type, txt
ORDER BY rank DESC
2) In the above query, why can't I write "HAVING rank > 0"? instead of
repeating the whole entire sum() expression "HAVING sum(...)"
Thanks in advance,
Alex Rice, Software Developer
Architectural Research Consultants, Inc.
alrice@ARCplanning.com
alrice@swcp.com
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: 2 simple SQL questions: optimizing aggegate query
2003-05-05 18:59 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
@ 2003-05-05 19:45 ` Bruno Wolff III <bruno@wolff.to>
2003-05-05 20:03 ` Re: 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
0 siblings, 1 reply; 5+ messages in thread
From: Bruno Wolff III @ 2003-05-05 19:45 UTC (permalink / raw)
To: Alex Rice <alrice@ARCplanning.com>; +Cc: pgsql-sql
On Mon, May 05, 2003 at 12:59:47 -0600,
Alex Rice <alrice@ARCplanning.com> wrote:
> Hello, I am using Mnogosearch with a PostgreSQL backend. I am writing
> my own queries because I can't use the existing Perl or PHP front ends
> to Mnogosearch.
>
> 1) Is there a way to optimize this query? It takes ~6 seconds on my
> workstation :-( I would like to get it under 2 seconds.
>
> SELECT url.rec_id, url, title, content_type, txt,
> sum(
> case
> when dict.word = 'wordx' then 1
> when dict.word = 'wordx' then 1
> when dict.word = 'wordy' then 1
> else 0
> end
> ) as rank
> FROM dict, url
> WHERE url.rec_id = dict.url_id
> GROUP BY rec_id, url, title, content_type, txt
> ORDER BY rank DESC
Is there some reason you can't put the word list in the where clause?
(You can use coalesce to set NULL sums to 0.)
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: 2 simple SQL questions: optimizing aggegate query
2003-05-05 18:59 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
2003-05-05 19:45 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
@ 2003-05-05 20:03 ` Alex Rice <alrice@ARCplanning.com>
2003-05-06 01:28 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
0 siblings, 1 reply; 5+ messages in thread
From: Alex Rice @ 2003-05-05 20:03 UTC (permalink / raw)
To: Bruno Wolff III <bruno@wolff.to>; +Cc: pgsql-sql
On Monday, May 5, 2003, at 01:45 PM, Bruno Wolff III wrote:
>> SELECT url.rec_id, url, title, content_type, txt,
>> sum(
>> case
>> when dict.word = 'wordx' then 1
>> when dict.word = 'wordx' then 1
>> when dict.word = 'wordy' then 1
>> else 0
>> end
>> ) as rank
>> FROM dict, url
>> WHERE url.rec_id = dict.url_id
>> GROUP BY rec_id, url, title, content_type, txt
>> ORDER BY rank DESC
>
> Is there some reason you can't put the word list in the where clause?
> (You can use coalesce to set NULL sums to 0.)
Can you give an example of what you mean use coalesce and how to use it
to produce a count of the matched words, the rank?
I am converting some existing query examples that use the form
SUM( dict.word IN ( word list...))
which doesn't work with pgsql.
I got the idea for the case statement from this archive message
http://archives.postgresql.org/pgsql-sql/1999-10/msg00136.php
Thanks much,
Alex Rice, Software Developer
Architectural Research Consultants, Inc.
alrice@ARCplanning.com
alrice@swcp.com
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: 2 simple SQL questions: optimizing aggegate query
2003-05-05 18:59 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
2003-05-05 19:45 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
2003-05-05 20:03 ` Re: 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
@ 2003-05-06 01:28 ` Bruno Wolff III <bruno@wolff.to>
2003-05-06 19:28 ` Re: 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
0 siblings, 1 reply; 5+ messages in thread
From: Bruno Wolff III @ 2003-05-06 01:28 UTC (permalink / raw)
To: Alex Rice <alrice@ARCplanning.com>; +Cc: pgsql-sql
On Mon, May 05, 2003 at 14:03:18 -0600,
Alex Rice <alrice@ARCplanning.com> wrote:
>
> On Monday, May 5, 2003, at 01:45 PM, Bruno Wolff III wrote:
>
> >>SELECT url.rec_id, url, title, content_type, txt,
> >>sum(
> >> case
> >> when dict.word = 'wordx' then 1
> >> when dict.word = 'wordx' then 1
> >> when dict.word = 'wordy' then 1
> >> else 0
> >> end
> >>) as rank
> >>FROM dict, url
> >>WHERE url.rec_id = dict.url_id
> >>GROUP BY rec_id, url, title, content_type, txt
> >>ORDER BY rank DESC
> >
> >Is there some reason you can't put the word list in the where clause?
> >(You can use coalesce to set NULL sums to 0.)
>
> Can you give an example of what you mean use coalesce and how to use it
> to produce a count of the matched words, the rank?
You don't really need coalesce. I mentioned that because if you take
the sum of zero rows (the case where no words match for one of the returned
rows) then you get NULL instead of 0 (which may or may not be a problem
for you).
> I am converting some existing query examples that use the form
> SUM( dict.word IN ( word list...))
> which doesn't work with pgsql.
You want to do this outside the sum, not inside (and change sum to
count). For example:
select count(dict.word) from dict where word in (word list ...);
(If word list is a select rather than a list of constants, you currently
want to use exists instead for performance reasons. In 7.4 that won't
be a problem.)
By putting things in the where close you don't have to examine all of
the rows for each word - url pair. This may cut out a lot of rows that
need to be examined if there is an index on dict.word and the number of
words in a url is large relative to the number being searched on.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: 2 simple SQL questions: optimizing aggegate query
2003-05-05 18:59 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
2003-05-05 19:45 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
2003-05-05 20:03 ` Re: 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
2003-05-06 01:28 ` Re: 2 simple SQL questions: optimizing aggegate query Bruno Wolff III <bruno@wolff.to>
@ 2003-05-06 19:28 ` Alex Rice <alrice@ARCplanning.com>
0 siblings, 0 replies; 5+ messages in thread
From: Alex Rice @ 2003-05-06 19:28 UTC (permalink / raw)
To: Bruno Wolff III <bruno@wolff.to>; +Cc: pgsql-sql
On Monday, May 5, 2003, at 07:28 PM, Bruno Wolff III wrote:
>> I am converting some existing query examples that use the form
>> SUM( dict.word IN ( word list...))
>> which doesn't work with pgsql.
>
> You want to do this outside the sum, not inside (and change sum to
> count). For example:
> select count(dict.word) from dict where word in (word list ...);
> (If word list is a select rather than a list of constants, you
> currently
> want to use exists instead for performance reasons. In 7.4 that won't
> be a problem.)
>
> By putting things in the where close you don't have to examine all of
> the rows for each word - url pair. This may cut out a lot of rows that
> need to be examined if there is an index on dict.word and the number of
> words in a url is large relative to the number being searched on.
>
Bruno: Thanks! Using your advice, my queries are now < 1 sec! Using
this form
SELECT url_id, url.url, url.txt, url.content_type,
url.docsize, count(dict.word) AS rank
FROM url,dict WHERE word IN ('word1', 'word2', ...)
AND url.rec_id = dict.url_id
GROUP BY url_id, url, txt, content_type, docsize
ORDER BY rank DESC;
I have also upgraded to pgsql 7.3.2, and the query I posted yesterday
"sum( case ... ) " is a lot faster as well (about 2 seconds now). So I
think maybe there was also something wrong with my indices yesterday-
in addition to there being something wrong with my SQL. :-)
Thanks for the SQL advice.
Alex Rice, Software Developer
Architectural Research Consultants, Inc.
alrice@ARCplanning.com
alrice@swcp.com
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2003-05-06 19:28 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2003-05-05 18:59 2 simple SQL questions: optimizing aggegate query Alex Rice <alrice@ARCplanning.com>
2003-05-05 19:45 ` Bruno Wolff III <bruno@wolff.to>
2003-05-05 20:03 ` Alex Rice <alrice@ARCplanning.com>
2003-05-06 01:28 ` Bruno Wolff III <bruno@wolff.to>
2003-05-06 19:28 ` Alex Rice <alrice@ARCplanning.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox