From: Heikki Linnakangas <heikki@enterprisedb.com>
To: Dimitri Fontaine <dfontaine@hi-media.com>
Cc: pgsql-performance@postgresql.org
Cc: Simon Riggs <simon@2ndquadrant.com>
Cc: Jignesh K. Shah <J.K.Shah@sun.com>
Cc: Greg Smith <gsmith@gregsmith.com>
Subject: Re: Benchmark Data requested
Date: Tue, 05 Feb 2008 20:06:17 +0000
Message-ID: <47A8C1B9.10200@enterprisedb.com> (raw)
In-Reply-To: <200802051815.25800.dfontaine@hi-media.com>
References: <1202150236.4252.521.camel@ebony.site>
<200802051506.51581.dfontaine@hi-media.com>
<1202221496.4252.680.camel@ebony.site>
<200802051815.25800.dfontaine@hi-media.com>
Dimitri Fontaine wrote:
> Le mardi 05 février 2008, Simon Riggs a écrit :
>> I'll look at COPY FROM internals to make this faster. I'm looking at
>> this now to refresh my memory; I already had some plans on the shelf.
>
> Maybe stealing some ideas from pg_bulkload could somewhat help here?
> http://pgfoundry.org/docman/view.php/1000261/456/20060709_pg_bulkload.pdf
>
> IIRC it's mainly about how to optimize index updating while loading data, and
> I've heard complaints on the line "this external tool has to know too much
> about PostgreSQL internals to be trustworthy as non-core code"... so...
I've been thinking of looking into that as well. The basic trick
pg_bulkload is using is to populate the index as the data is being
loaded. There's no fundamental reason why we couldn't do that internally
in COPY. Triggers or constraints that access the table being loaded
would make it impossible, but we should be able to detect that and fall
back to what we have now.
What I'm basically thinking about is to modify the indexam API of
building a new index, so that COPY would feed the tuples to the indexam,
instead of the indexam opening and scanning the heap. The b-tree indexam
would spool the tuples into a tuplesort as the COPY progresses, and
build the index from that at the end as usual.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: heikki@enterprisedb.com, dfontaine@hi-media.com, simon@2ndquadrant.com, J.K.Shah@sun.com, gsmith@gregsmith.com
Subject: Re: Benchmark Data requested
In-Reply-To: <47A8C1B9.10200@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox