From: Carlo Stonebanks <stonec.register@sympatico.ca>
To: pgsql-performance@postgresql.org
Subject: Re: Performance Optimization for Dummies 2 - the SQL
Date: Fri, 6 Oct 2006 16:45:07 -0400
Message-ID: <eg6f8l$2det$1@news.hub.org> (raw)
References: <eft782$5a1$1@news.hub.org>
<b42b73150610031221y4955e663ped999a83a810d318@mail.gmail.com>
<eg0u6e$hbr$1@news.hub.org>
<b42b73150610041407y3554f311u1329c4f3bdc53999@mail.gmail.com>
<eg1dai$1tfa$1@news.hub.org>
<b42b73150610042326v7fdeed17rf6b96e52a681a6db@mail.gmail.com>
<eg2cds$29en$1@news.hub.org>
<b42b73150610050630t76e9eedeh45a382729d35f2f6@mail.gmail.com>
<eg611q$l0m$1@news.hub.org>
<1160153623.6181.13.camel@state.g2switchworks.com>
<b42b73150610061153i3d11d7a4r23e1d7c5317feb9f@mail.gmail.com>
> how did you determine that it is done every 500 rows? this is the
The import program pages the import table - it is currently set at 500 rows
per page. With each page, I run an ANALYZE.
> default autovacuum paramater. if you followed my earlier
> recommendations, you are aware that autovacuum (which also analyzes)
> is not running during bulk inserts, right?
It's intuitivly obvious, but I can't do bulk inserts. It's just not the
nature of what we are doing with the data.
> imo, best way to do big data import/conversion is to:
> 1. turn off all extra features, like stats, logs, etc
done
> 2. use copy interface to load data into scratch tables with probably
> all text fields
done
> 3. analyze (just once)
I think this doesn't apply in our case, because we aren't doing bulk
inserts.
> 4. use big queries to transform, normalize, etc
This is currently being done programmatically. The nature of what we're
doing is suited for imperitive, navigational logic rather than declarative,
data set logic; just the opposite of what SQL likes, I know! If there's some
way to replace thousands of lines of analysis and decision trees with
ultrafast queries - great...
> important feature of analyze is to tell the planner approx. how big
> the tables are.
But the tables grow as the process progresses - would you not want the
server to re-evaluate its strategy periodically?
Carlo
>
> merlin
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: stonec.register@sympatico.ca
Subject: Re: Performance Optimization for Dummies 2 - the SQL
In-Reply-To: <eg6f8l$2det$1@news.hub.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox