From: Shane Ambler <pgsql@Sheeky.Biz>
To: John Rouillard <rouilj@renesys.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Very poor performance loading 100M of sql data using copy
Date: Tue, 29 Apr 2008 05:19:59 +0930
Message-ID: <48162A67.3090500@Sheeky.Biz> (raw)
In-Reply-To: <20080428180053.GL6622@renesys.com>
References: <20080428172431.GI6622@renesys.com>
<48160F05.2090207@enterprisedb.com>
<20080428180053.GL6622@renesys.com>
John Rouillard wrote:
> We can't do this as we are backfilling a couple of months of data
> into tables with existing data.
Is this a one off data loading of historic data or an ongoing thing?
>>> The only indexes we have to drop are the ones on the primary keys
>>> (there is one non-primary key index in the database as well).
If this amount of data importing is ongoing then one thought I would try
is partitioning (this could be worthwhile anyway with the amount of data
you appear to have).
Create an inherited table for the month being imported, load the data
into it, then add the check constraints, indexes, and modify the
rules/triggers to handle the inserts to the parent table.
--
Shane Ambler
pgSQL (at) Sheeky (dot) Biz
Get Sheeky @ http://Sheeky.Biz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: pgsql@Sheeky.Biz, rouilj@renesys.com
Subject: Re: Very poor performance loading 100M of sql data using copy
In-Reply-To: <48162A67.3090500@Sheeky.Biz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox