X-Original-To: pgsql-sql-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id 5CCB4D1B897 for ; Fri, 6 Feb 2004 03:16:26 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 10569-02 for ; Thu, 5 Feb 2004 23:16:21 -0400 (AST) Received: from server146.hostingrevolution.com (server146.hostingrevolution.com [64.21.143.245]) by svr1.postgresql.org (Postfix) with ESMTP id 2DE0ED1D141 for ; Thu, 5 Feb 2004 23:16:16 -0400 (AST) Received: from [144.138.233.118] (helo=athlon2000) by server146.hostingrevolution.com with smtp (Exim 4.24) id 1AowTU-0002x3-DN for pgsql-sql@postgresql.org; Thu, 05 Feb 2004 21:15:51 -0600 Message-ID: <003b01c3ec5f$d740c3e0$76e98a90@athlon2000> From: "Richard Sydney-Smith" To: Subject: Re: Slow sub-selects, max and count(*) Date: Thu, 5 Feb 2004 15:53:08 +0800 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0018_01C3EC00.262C2BD0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server146.hostingrevolution.com X-AntiAbuse: Original Domain - postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12] X-AntiAbuse: Sender Address Domain - ibisaustralia.com X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200402/58 X-Sequence-Number: 16749 This is a multi-part message in MIME format. ------=_NextPart_000_0018_01C3EC00.262C2BD0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thanks Josh and Ian, narrowing the problem down. The really slow line is the one where I try and= remove potential duplicates. It does not look at the indexes.=20 Point on using copy rather than insert is taken. I use copy to load to a te= mp file so I can test the data and alter certain values. I suppose I could:= copy to temp / alter/ save/ copy into fsechist but that is rather ugly and= as it is not the insert that is the real bug bear I would like to fix this= first. Ian you suggested a simular problem was solved using "exists" rather than "= in". I am not sure how the use differs. I have tried to include it in optio= n 4 below.=20 Trial Solutions: 1) delete from fsechist where hist_q in (select hist_q from fsechist, temp_= shareprices where hist_tick =3D ticker and dte =3D hist_date); far too slow 2)=20 update fsechist set hist_tick =3D \'@@\' from fsechist h, temp_shareprices= t where t.ticker =3D h.hist_tick and h.hist_date =3D t.dte; delete from fsechist where hist_tick =3D \'@@\'; really , really slow. Thought this would use the indexes but "explain" ( be= ing my friend) still gives the query plan as a sequential scan 3)=20 -- does not allow insert of a single company data delete from fsechist where hist_date in (select distinct dte from temp_shar= eprices); Works quick enough but has limited functionality. ( does not cope with impo= rts other than a full end of trading) 4)=20 delete from fsechist where exists(select 1 from fsechist, temp_shareprices = where hist_tick =3D ticker and dte =3D hist_date); Tried and still took over 60 secs before I cancelled the request. Indexes Both tables have indexes defined for tick and date. tick and date of the sa= me data type in both cases. ------=_NextPart_000_0018_01C3EC00.262C2BD0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Thanks Josh and Ian,
 
narrowing the problem down. The really&nbs= p;slow=20 line is the one where I try and remove potential duplicates. It does not lo= ok at=20 the indexes.
 
Point on using copy rather than insert is = taken. I=20 use copy to load to a temp file so I can test the data and alter certain va= lues.=20 I suppose I could: copy to temp / alter/ save/ copy into fsechist= but=20 that is rather ugly and as it is not the insert that is the real bug bear I= =20 would like to fix this first.
 
Ian you suggested a simular problem was so= lved=20 using "exists" rather than "in". I am not sure how the use differs. I have = tried=20 to include it in option 4 below.
 
Trial Solutions:
 
1) delete from fsechist where hist_q = in=20 (select hist_q from fsechist, temp_shareprices where hist_tick =3D ticker a= nd dte=20 =3D hist_date);
far too slow
 
2) 
 
 update fsechist set hist_tick =3D \'= @@\' from=20 fsechist h, temp_shareprices t where t.ticker =3D h.hist_tick and h.hist_da= te =3D=20 t.dte;
 delete from fsechist whe= re=20 hist_tick =3D \'@@\';
 
really , really slow. Thought this would u= se the=20 indexes but "explain" ( being my friend) still gives the query plan as a=20 sequential scan
 
3)
 
-- does not allow insert of a single compa= ny=20 data
delete from fsechist where hist_date in (select distinct dte from= =20 temp_shareprices);
 
Works quick enough but has limited functio= nality. (=20 does not cope with imports other than a full end of trading)
 
4)
 
delete from fsechist where exists(select 1= from=20 fsechist, temp_shareprices where hist_tick =3D ticker and dte =3D=20 hist_date);
 
Tried and still took over 60 secs before I= =20 cancelled the request.
Indexes
 
Both tables have indexes defined for tick = and date.=20 tick and date of the same data type in both cases.
 
------=_NextPart_000_0018_01C3EC00.262C2BD0--