Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 211409FA4EE for ; Fri, 29 Sep 2006 01:37:43 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 60124-02 for ; Fri, 29 Sep 2006 01:37:38 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from floppy.pyrenet.fr (news.pyrenet.fr [194.116.145.2]) by postgresql.org (Postfix) with ESMTP id 786EB9FA432 for ; Fri, 29 Sep 2006 01:37:38 -0300 (ADT) Received: by floppy.pyrenet.fr (Postfix, from userid 106) id 8D30330E2B; Fri, 29 Sep 2006 06:37:36 +0200 (MET DST) From: "Carlo Stonebanks" X-Newsgroups: pgsql.performance Subject: Re: Performace Optimization for Dummies Date: Fri, 29 Sep 2006 00:37:37 -0400 Organization: Hub.Org Networking Services Lines: 15 Message-ID: References: <20060928184538.GV34238@nasby.net> X-Complaints-To: usenet@news.hub.org X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962 X-RFC2646: Format=Flowed; Response To: pgsql-performance@postgresql.org X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.071 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200609/415 X-Sequence-Number: 21101 > imo, the key to high performance big data movements in postgresql is > mastering sql and pl/pgsql, especially the latter. once you get good > at it, your net time of copy+plpgsql is going to be less than > insert+tcl. If this implies bulk inserts, I'm afraid I have to consider something else. Any data that has been imported and dedpulicated has to be placed back into the database so that it can be available for the next imported row (there are currently 16 tables affected, and more to come). If I was to cache all inserts into a seperate resource, then I would have to search 32 tables - the local pending resources, as well as the data still in the system. I am not even mentioning that imports do not just insert rows, they could just rows, adding their own complexity.