Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 047769FB2D4 for ; Fri, 6 Oct 2006 17:45:23 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 89534-04 for ; Fri, 6 Oct 2006 20:45:17 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from floppy.pyrenet.fr (floppy.pyrenet.fr [194.116.145.2]) by postgresql.org (Postfix) with ESMTP id EC33F9FB2C6 for ; Fri, 6 Oct 2006 17:45:16 -0300 (ADT) Received: by floppy.pyrenet.fr (Postfix, from userid 106) id 7599B308E6; Fri, 6 Oct 2006 22:45:15 +0200 (MET DST) From: "Carlo Stonebanks" X-Newsgroups: pgsql.performance Subject: Re: Performance Optimization for Dummies 2 - the SQL Date: Fri, 6 Oct 2006 16:45:07 -0400 Organization: Hub.Org Networking Services Lines: 53 Message-ID: References: <1160153623.6181.13.camel@state.g2switchworks.com> X-Complaints-To: usenet@news.hub.org X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.2869 X-RFC2646: Format=Flowed; Response X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962 To: pgsql-performance@postgresql.org X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200610/138 X-Sequence-Number: 21267 > how did you determine that it is done every 500 rows? this is the The import program pages the import table - it is currently set at 500 rows per page. With each page, I run an ANALYZE. > default autovacuum paramater. if you followed my earlier > recommendations, you are aware that autovacuum (which also analyzes) > is not running during bulk inserts, right? It's intuitivly obvious, but I can't do bulk inserts. It's just not the nature of what we are doing with the data. > imo, best way to do big data import/conversion is to: > 1. turn off all extra features, like stats, logs, etc done > 2. use copy interface to load data into scratch tables with probably > all text fields done > 3. analyze (just once) I think this doesn't apply in our case, because we aren't doing bulk inserts. > 4. use big queries to transform, normalize, etc This is currently being done programmatically. The nature of what we're doing is suited for imperitive, navigational logic rather than declarative, data set logic; just the opposite of what SQL likes, I know! If there's some way to replace thousands of lines of analysis and decision trees with ultrafast queries - great... > important feature of analyze is to tell the planner approx. how big > the tables are. But the tables grow as the process progresses - would you not want the server to re-evaluate its strategy periodically? Carlo > > merlin > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org >