Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 519B49FB2F4 for ; Tue, 3 Oct 2006 00:03:52 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 41660-08 for ; Tue, 3 Oct 2006 00:03:45 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from floppy.pyrenet.fr (floppy.pyrenet.fr [194.116.145.2]) by postgresql.org (Postfix) with ESMTP id 731BE9FB2DB for ; Tue, 3 Oct 2006 00:03:45 -0300 (ADT) Received: by floppy.pyrenet.fr (Postfix, from userid 106) id B728430E1A; Tue, 3 Oct 2006 05:03:43 +0200 (MET DST) From: "Carlo Stonebanks" X-Newsgroups: pgsql.performance Subject: Re: Performace Optimization for Dummies Date: Mon, 2 Oct 2006 23:03:31 -0400 Organization: Hub.Org Networking Services Lines: 13 Message-ID: References: <451D0F67.4040309@logix-tt.com> X-Complaints-To: usenet@news.hub.org X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962 X-RFC2646: Format=Flowed; Original To: pgsql-performance@postgresql.org X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.066 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200610/17 X-Sequence-Number: 21146 > Did you think about putting the whole data into PostgreSQL using COPY in > a nearly unprocessed manner, index it properly, and then use SQL and > stored functions to transform the data inside the database to the > desired result? This is actually what we are doing. The slowness is on the row-by-row transformation. Every row reqauires that all the inserts and updates of the pvious row be committed - that's why we have problems figuring out how to use this using SQL set logic. Carlo