To: Neil Conway <neilc@samurai.com>
cc: Hannu Krosing <hannu@skype.net>, Josh Berkus <josh@agliodbs.com>,
	pgsql-hackers@postgresql.org, Simon Riggs <simon@2ndquadrant.com>
Subject: Re: VLDB Features 
In-reply-to: <1197660423.1536.8.camel@dell.linuxdev.us.dell.com> 
References: <1197371508.4255.1327.camel@ebony.site>
	<200712111053.11178.josh@agliodbs.com>
	<1197416461.29432.10.camel@dell.linuxdev.us.dell.com>
	<1197636530.7974.16.camel@hannu-laptop>
	<1197660423.1536.8.camel@dell.linuxdev.us.dell.com>
Comments: In-reply-to Neil Conway <neilc@samurai.com>
	message dated "Fri, 14 Dec 2007 11:27:03 -0800"
Date: Fri, 14 Dec 2007 18:22:41 -0500
Message-ID: <15007.1197674561@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>

Neil Conway <neilc@samurai.com> writes:
> By modifying COPY: COPY IGNORE ERRORS or some such would instruct COPY
> to drop (and log) rows that contain malformed data. That is, rows with
> too many or too few columns, rows that result in constraint violations,
> and rows containing columns where the data type's input function raises
> an error. The last case is the only thing that would be a bit tricky to
> implement, I think: you could use PG_TRY() around the InputFunctionCall,
> but I guess you'd need a subtransaction to ensure that you reset your
> state correctly after catching an error.

Yeah.  It's the subtransaction per row that's daunting --- not only the
cycles spent for that, but the ensuing limitation to 4G rows imported
per COPY.

If we could somehow only do a subtransaction per failure, things would
be much better, but I don't see how.

			regards, tom lane