Received: from localhost (maia-5.hub.org [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id AF4BB9FA372 for ; Sat, 23 Dec 2006 16:44:13 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-new, port 10024) with ESMTP id 75542-08 for ; Sat, 23 Dec 2006 16:44:01 -0400 (AST) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.4 Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.231]) by postgresql.org (Postfix) with ESMTP id 3ABF49FA363 for ; Sat, 23 Dec 2006 16:44:03 -0400 (AST) Received: by nz-out-0506.google.com with SMTP id i11so1381443nzh for ; Sat, 23 Dec 2006 12:44:02 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=g1Lu3vRK7eIHvnRLq84my85sBJcyck6HbvkD+csPbIIXPlwbo3iY0PcGqIDkmCzeveNlUBSUxXZKN9ufTJlFt2zoe1FcUJ71P3XOS116fI07La3fw8ALWhIz9rqvdKL0KsbOI6/Ayp6u3pjm6ntTL1SzAOubx5cTQ1l/lDGJ2zw= Received: by 10.65.147.1 with SMTP id z1mr2675114qbn.1166906641535; Sat, 23 Dec 2006 12:44:01 -0800 (PST) Received: by 10.65.206.17 with HTTP; Sat, 23 Dec 2006 12:44:01 -0800 (PST) Message-ID: Date: Sun, 24 Dec 2006 01:44:01 +0500 From: "Shoaib Mir" To: "Benjamin Arai" Subject: Re: [GENERAL] OUTER JOIN IS SLOW Cc: pgsql-general@postgresql.org, pgsql-admin@postgresql.org In-Reply-To: <458D9437.1000609@araisoft.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_45751_1533285.1166906641465" References: <458D8D56.6020506@araisoft.com> <458D9437.1000609@araisoft.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200612/311 X-Sequence-Number: 24103 ------=_Part_45751_1533285.1166906641465 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Yes, ANALYZE should definitely improve the performance for query... -------------- Shoaib Mir EnterpriseDB (www.enterprisedb.com) On 12/24/06, Benjamin Arai wrote: > > Just to make things more clear I ran EXPLAIN ANALYZE on the slow query. > I got > > Merge Full Join (cost=62.33..73.36 rows=1000 width=19) (actual > time=39.205..8521.644 rows=272 loops=1) > Merge Cond: ("outer".pkey = "inner".d1) > -> Index Scan using mutualfd_weekday_qbid_pkey_idx on > mutualfd_weekday_qbid (cost=0.00..6.01 rows=1 width=19) (actual > time=34.579..8510.801 rows=253 loops=1) > Index Cond: ((pkey >= '2005-12-15'::date) AND (pkey <= > '2006-12-15'::date)) > Filter: (cusip = '92193920'::text) > -> Sort (cost=62.33..64.83 rows=1000 width=4) (actual > time=4.604..5.674 rows=262 loops=1) > Sort Key: d1.d1 > -> Function Scan on "getDateRange" d1 (cost=0.00..12.50 > rows=1000 width=4) (actual time=2.591..3.591 rows=262 loops=1) > Total runtime: 8522.894 ms > (9 rows) > > It looks like this line: > > "-> Index Scan using mutualfd_weekday_qbid_pkey_idx on > mutualfd_weekday_qbid (cost=0.00..6.01 rows=1 width=19) (actual > time=34.579..8510.801 rows=253 loops=1)" > > is the problem. As I understand it is using the index but it is low as > dirt. Hopefully, the -z will fix this. I also ran the same query but > with earlier dates in the table and the query ran much faster. > > Merge Full Join (cost=3492.48..3505.60 rows=1034 width=19) (actual > time=7.605..12.851 rows=273 loops=1) > Merge Cond: ("outer".d1 = "inner".pkey) > -> Sort (cost=62.33..64.83 rows=1000 width=4) (actual > time=4.586..5.530 rows=263 loops=1) > Sort Key: d1.d1 > -> Function Scan on "getDateRange" d1 (cost=0.00..12.50 > rows=1000 width=4) (actual time=2.493..3.515 rows=263 loops=1) > -> Sort (cost=3430.15..3432.74 rows=1034 width=19) (actual > time=2.998..3.971 rows=263 loops=1) > Sort Key: mutualfd_weekday_qbid.pkey > -> Index Scan using mutualfd_weekday_qbid_pkey on > mutualfd_weekday_qbid (cost=0.00..3378.38 rows=1034 width=19) (actual > time=0.075..1.843 rows=263 loops=1) > Index Cond: ((cusip = '92193920'::text) AND (pkey >= > '1999-12-15'::date) AND (pkey <= '2000-12-15'::date)) > Total runtime: 13.935 ms > (10 rows) > > In this case it only took 13.935ms as compared to 8522.894 ms for the > newer data. > > Benjamin > > Shoaib Mir wrote: > > ....adding to the last email, for now try the work_mem but you should > > be adding ANALYZE along with the VACUUM (with a cron job I guess) you > > do regularly. > > > > ------------ > > Shoaib Mir > > EntperpriseDB ( www.enterprisedb.com ) > > > > On 12/24/06, *Shoaib Mir* > > wrote: > > > > Try increasing the work_mem first to see the change, that might > help. > > > > ------------- > > Shoaib Mir > > EnterpriseDB ( www.enterprisedb.com ) > > > > On 12/24/06, *Benjamin Arai* > > wrote: > > > > I have been running pieces of my PL function by hand and I > > have found > > that the following queries work by themselves taking less than > > a second > > to execute. > > > > getDateRange"('12/1/2005','12/1/2006') <- simply generates a > > date > > list. Doesn't even access a table > > > > SELECT * FROM mutualfd_weekday_qbid WHERE cusip='92193920' > > AND pkey >= > > '12/15/2005' AND pkey <= '12/15/2006'; > > > > But when combined as below it takes 10 seconds to execute. > > > > SELECT d1 as date, d2.data as data FROM > > "getDateRange"('12/1/2005','12/1/2006') d1 FULL OUTER JOIN > > (SELECT * > > FROM mutualfd_weekday_qbid WHERE cusip='92193920' AND pkey >= > > '12/15/2005' AND pkey <= '12/15/2006') d2 ON d1=d2.pkey; > > > > Do I need to increase the work_mem or is this possible still a > > ANALYZE > > issue? > > > > Benjamin > > > > ---------------------------(end of > > broadcast)--------------------------- > > TIP 7: You can help support the PostgreSQL project by donating > at > > > > http://www.postgresql.org/about/donate > > > > > > > > > ------=_Part_45751_1533285.1166906641465 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Yes, ANALYZE should definitely improve the performance for query...

--------------
Shoaib Mir
EnterpriseDB (www.enterprisedb.com)

On 12/24/06, Benjamin Arai <benjamin@araisoft.com> wrote:
Just to make things more clear I ran EXPLAIN ANALYZE on the slow query.
I got

Merge Full Join  (cost=62.33..73.36 rows=1000 width=19) (actual
time=39.205..8521.644 rows=272 loops=1)
   Merge Cond: ("outer".pkey = "inner".d1)
   ->  Index Scan using mutualfd_weekday_qbid_pkey_idx on
mutualfd_weekday_qbid  (cost=0.00..6.01 rows=1 width=19) (actual
time=34.579..8510.801 rows=253 loops=1)
         Index Cond: ((pkey >= '2005-12-15'::date) AND (pkey <=
'2006-12-15'::date))
         Filter: (cusip = '92193920'::text)
   ->  Sort  (cost=62.33..64.83 rows=1000 width=4) (actual
time=4.604..5.674 rows=262 loops=1)
         Sort Key: d1.d1
         ->  Function Scan on "getDateRange" d1  (cost=0.00..12.50
rows=1000 width=4) (actual time=2.591..3.591 rows=262 loops=1)
Total runtime: 8522.894 ms
(9 rows)

It looks like this line:

"->  Index Scan using mutualfd_weekday_qbid_pkey_idx on
mutualfd_weekday_qbid  (cost=0.00..6.01 rows=1 width=19) (actual
time=34.579..8510.801 rows=253 loops=1)"

is the problem.  As I understand it is using the index but it is low as
dirt.  Hopefully, the -z will fix this.  I also ran the same query but
with earlier dates in the table and the query ran much faster.

Merge Full Join  (cost=3492.48..3505.60 rows=1034 width=19) (actual
time= 7.605..12.851 rows=273 loops=1)
   Merge Cond: ("outer".d1 = "inner".pkey)
   ->  Sort  (cost=62.33..64.83 rows=1000 width=4) (actual
time=4.586..5.530 rows=263 loops=1)
         Sort Key: d1.d1
         ->  Function Scan on "getDateRange" d1  (cost=0.00..12.50
rows=1000 width=4) (actual time=2.493..3.515 rows=263 loops=1)
   ->  Sort  (cost=3430.15..3432.74 rows=1034 width=19) (actual
time=2.998..3.971 rows=263 loops=1)
         Sort Key: mutualfd_weekday_qbid.pkey
         ->  Index Scan using mutualfd_weekday_qbid_pkey on
mutualfd_weekday_qbid  (cost=0.00..3378.38 rows=1034 width=19) (actual
time=0.075..1.843 rows=263 loops=1)
               Index Cond: ((cusip = '92193920'::text) AND (pkey >=
'1999-12-15'::date) AND (pkey <= '2000-12-15'::date))
Total runtime: 13.935 ms
(10 rows)

In this case it only took 13.935ms as compared to 8522.894 ms for the
newer data.

Benjamin

Shoaib Mir wrote:
> ....adding to the last email, for now try the work_mem but you should
> be adding ANALYZE along with the VACUUM (with a cron job I guess) you
> do regularly.
>
> ------------
> Shoaib Mir
> EntperpriseDB ( www.enterprisedb.com <http://www.enterprisedb.com>)
>
> On 12/24/06, *Shoaib Mir* <shoaibmir@gmail.com
> <mailto: shoaibmir@gmail.com>> wrote:
>
>     Try increasing the work_mem first to see the change, that might help.
>
>     -------------
>     Shoaib Mir
>     EnterpriseDB ( www.enterprisedb.com <http://www.enterprisedb.com>)
>
>     On 12/24/06, *Benjamin Arai* <benjamin@araisoft.com
>     <mailto:benjamin@araisoft.com>> wrote:
>
>         I have been running pieces of my PL function by hand and I
>         have found
>         that the following queries work by themselves taking less than
>         a second
>         to execute.
>
>         getDateRange"('12/1/2005','12/1/2006')  <- simply generates a
>         date
>         list.  Doesn't even access a table
>
>         SELECT * FROM mutualfd_weekday_qbid WHERE cusip='92193920'
>         AND  pkey >=
>         '12/15/2005' AND pkey <= '12/15/2006';
>
>         But when combined as below it takes 10 seconds to execute.
>
>         SELECT d1 as date, d2.data as data FROM
>         "getDateRange"('12/1/2005','12/1/2006') d1 FULL OUTER JOIN
>         (SELECT *
>         FROM mutualfd_weekday_qbid WHERE cusip='92193920' AND  pkey >=
>         '12/15/2005' AND pkey <= '12/15/2006') d2 ON d1=d2.pkey;
>
>         Do I need to increase the work_mem or is this possible still a
>         ANALYZE
>         issue?
>
>         Benjamin
>
>         ---------------------------(end of
>         broadcast)---------------------------
>         TIP 7: You can help support the PostgreSQL project by donating at
>
>                         http://www.postgresql.org/about/donate
>         <http://www.postgresql.org/about/donate >
>
>
>

------=_Part_45751_1533285.1166906641465--