Subject: Re: VLDB Features
From: Neil Conway <neilc@samurai.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Hannu Krosing <hannu@skype.net>, Josh Berkus <josh@agliodbs.com>,
	pgsql-hackers@postgresql.org, Simon Riggs <simon@2ndquadrant.com>
In-Reply-To: <15007.1197674561@sss.pgh.pa.us>
References: <1197371508.4255.1327.camel@ebony.site>
	<200712111053.11178.josh@agliodbs.com>
	<1197416461.29432.10.camel@dell.linuxdev.us.dell.com>
	<1197636530.7974.16.camel@hannu-laptop>
	<1197660423.1536.8.camel@dell.linuxdev.us.dell.com>
	<15007.1197674561@sss.pgh.pa.us>
Content-Type: text/plain
Date: Fri, 14 Dec 2007 16:18:50 -0800
Message-Id: <1197677930.1536.18.camel@dell.linuxdev.us.dell.com>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Fri, 2007-12-14 at 18:22 -0500, Tom Lane wrote:
> If we could somehow only do a subtransaction per failure, things would
> be much better, but I don't see how.

One approach would be to essentially implement the pg_bulkloader
approach inside the backend. That is, begin by doing a subtransaction
for every k rows (with k = 1000, say). If you get any errors, then
either repeat the process with k/2 until you locate the individual
row(s) causing the trouble, or perhaps just immediately switch to k = 1.
Fairly ugly though, and would be quite slow for data sets with a high
proportion of erroneous data.

Another approach would be to distinguish between errors that require a
subtransaction to recover to a consistent state, and less serious errors
that don't have this requirement (e.g. invalid input to a data type
input function). If all the errors that we want to tolerate during a
bulk load fall into the latter category, we can do without
subtransactions.

-Neil