X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (neptune.hub.org [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id 39AF7D1B482 for ; Thu, 4 Dec 2003 22:46:06 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 37999-07 for ; Thu, 4 Dec 2003 18:45:36 -0400 (AST) Received: from anchor-post-33.mail.demon.net (anchor-post-33.mail.demon.net [194.217.242.91]) by svr1.postgresql.org (Postfix) with ESMTP id DD90FD1B46E for ; Thu, 4 Dec 2003 18:45:33 -0400 (AST) Received: from mwynhau.demon.co.uk ([193.237.186.96] helo=mainbox.archonet.com) by anchor-post-33.mail.demon.net with esmtp (Exim 3.35 #1) id 1AS2Dz-000FAX-0X; Thu, 04 Dec 2003 22:45:33 +0000 Received: by mainbox.archonet.com (Postfix, from userid 529) id 9775816CB0; Thu, 4 Dec 2003 22:45:25 +0000 (GMT) Received: from client17.archonet.com (client17.archonet.com [192.168.1.17]) by mainbox.archonet.com (Postfix) with ESMTP id 0442B16C65; Thu, 4 Dec 2003 22:45:22 +0000 (GMT) From: Richard Huxton To: Ivar Zarans , pgsql-performance@postgresql.org Subject: Re: Slow UPADTE, compared to INSERT Date: Thu, 4 Dec 2003 22:45:21 +0000 User-Agent: KMail/1.5 References: <20031204185751.GA15893@alcaron.ee> <200312042023.37050.dev@archonet.com> <20031204221312.GA18874@alcaron.ee> In-Reply-To: <20031204221312.GA18874@alcaron.ee> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200312042245.21899.dev@archonet.com> X-Bogosity: No, tests=bogofilter, spamicity=0.000000, version=0.15.3 X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200312/107 X-Sequence-Number: 4967 On Thursday 04 December 2003 22:13, Ivar Zarans wrote: > On Thu, Dec 04, 2003 at 08:23:36PM +0000, Richard Huxton wrote: > > Ah - it's probably not the update but the IN. You can rewrite it using > > PG's non-standard FROM: > > > > UPDATE t1 SET status='done' FROM t_tmp WHERE t1.rec_id = t_tmp.rec_id; > > This was one *very useful* hint! Using this method i got my processing > time of 24000 records down to around 3 minutes 10 seconds. Comparing > with initial 1 hour 20 minutes and then 16 minutes, this is impressive > improvement! Be aware, this is specific to PG - I'm not aware of this construction working on any other DB. Three minutes still doesn't sound brilliant, but that could be tuning issues. > > Now that doesn't explain why the update is taking so long. One fifth of a > > second is extremely slow. Are you certain that the index is being used? > > I posted results of "EXPLAIN" in my previous message. Meanwhile i tried > to update just one record, using "psql". Also tried out "EXPLAIN > ANALYZE". This way i did not see any big delay - total runtime for one > update was around 1 msec. Yep - the explain looked fine. If you run EXPLAIN ANALYSE it will give you timings too (actual timings will be slightly less than reported ones since PG won't be timing/reporting). > I am confused - has slowness of UPDATE something to do with Python and > PyPgSQL, since "psql" seems to have no delay whatsoever? Or is this > related to using two cursors, one for select results and other for > update? Even if this is related to Python or cursors, how am i getting > so big speed improvement only by using different query? Hmm - you didn't mention cursors. If this was a problem with PyPgSQL in general I suspect we'd know about it by now. It could however be some cursor-related issue. In general, you're probably better off trying to do updates/inserts as a single statement and letting PG manage things rather than processing one row at a time. If you've got the time, try putting together a small test-script with some dummy data and see if it's reproducible. I'm sure the other Python users would be interested in seeing where the problem is. -- Richard Huxton Archonet Ltd