From: Richard Huxton <dev@archonet.com>
To: frank church <pgsql@adontendev.net>
Cc: pgsql-sql@postgresql.org
Subject: Re: Loading lots of data in a SQL command
Date: Wed, 04 Jan 2006 09:21:22 +0000
Message-ID: <43BB9392.5080805@archonet.com> (raw)
In-Reply-To: <1136338331.43bb259b9334f@webmail.adontendev.net>
References: <1136338331.43bb259b9334f@webmail.adontendev.net>
frank church wrote:
> I am load lots of data via SQL into a database and wrapping it into transactions
> speeds it up.
>
> However this fails a number of times. The queries results are logged so it is
> easy for me to find problem records.
>
> However a single failure causes the whole transaction to fail.
>
> Is there a setting or feature that allows which allows the same performance as
> transactions, without causing the whole process to fail, like a delayed updates
> or write mechanism of some sort.
Not as it stands. I tend to use a small perl wrapper myself that loads
in batches of e.g. 10000 rows and if there is an error deal with it
separately.
I seem to recall it being discussed as a built-in feature recently
though, so there might be someone working on it for a future version.
> It is something I would like to set in that particular data looad.
You might find the "pgloader" project meets your needs exactly:
http://pgfoundry.org/projects/pgloader/
--
Richard Huxton
Archonet Ltd
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: dev@archonet.com, pgsql@adontendev.net
Subject: Re: Loading lots of data in a SQL command
In-Reply-To: <43BB9392.5080805@archonet.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox