DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references;
	b=Goy0sfvAAkXdvnEhDgOHHRUSIRPsmwJhUNik/o+GK1XcrTTErGQryJmOfxJ0NlM1AO6LT5P0hm1d0vFcjzFjUlTSic7Ztpz+gMe4t0saK+MvEol3bJcsI+v4euCWAncQ1oKYPirCUCrvesNdiqiKUEs9EAmYGw7wOX80YGrTfcA=
Message-ID: <d3c4af540712142332m4bd5f309k967f25e403c358bf@mail.gmail.com>
Date: Sat, 15 Dec 2007 13:02:51 +0530
From: NikhilS <nikkhils@gmail.com>
To: "Neil Conway" <neilc@samurai.com>
Subject: Re: VLDB Features
Cc: "Tom Lane" <tgl@sss.pgh.pa.us>, "Hannu Krosing" <hannu@skype.net>,
	"Josh Berkus" <josh@agliodbs.com>, pgsql-hackers@postgresql.org,
	"Simon Riggs" <simon@2ndquadrant.com>
In-Reply-To: <1197677930.1536.18.camel@dell.linuxdev.us.dell.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_4228_10971707.1197703971110"
References: <1197371508.4255.1327.camel@ebony.site>
	<200712111053.11178.josh@agliodbs.com>
	<1197416461.29432.10.camel@dell.linuxdev.us.dell.com>
	<1197636530.7974.16.camel@hannu-laptop>
	<1197660423.1536.8.camel@dell.linuxdev.us.dell.com>
	<15007.1197674561@sss.pgh.pa.us>
	<1197677930.1536.18.camel@dell.linuxdev.us.dell.com>

------=_Part_4228_10971707.1197703971110
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,

>
> Another approach would be to distinguish between errors that require a
> subtransaction to recover to a consistent state, and less serious errors
> that don't have this requirement (e.g. invalid input to a data type
> input function). If all the errors that we want to tolerate during a
> bulk load fall into the latter category, we can do without
> subtransactions.
>

I think errors which occur after we have done a fast_heap_insert of the
tuple generated from the current input row are the ones which would require
the subtransaction to recover. Examples could be unique/primary key
violation errors or FKey/triggers related errors. Any errors which occur
before doing the heap_insert should not require any recovery according to
me.

The overhead of having a subtransaction per row is a very valid concern. But
instead of using a per insert or a batch insert substraction, I am
thinking that we can start off a subtraction and continue it till we
encounter a failure. The moment an error is encountered, since we have the
offending (already in heap) tuple around, we can call a simple_heap_delete
on the same and commit (instead of aborting) this subtransaction after doing
some minor cleanup. This current input data row can also be logged into a
bad file. Recall that we need to only handle those errors in which the
simple_heap_insert is successful, but the index insertion or the after row
insert trigger causes an error. The rest of the load then can go ahead with
the start of a new subtransaction.

Regards,
Nikhils
-- 
EnterpriseDB               http://www.enterprisedb.com

------=_Part_4228_10971707.1197703971110
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi, <br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><br>Another approach would be to distinguish between errors that require a<br>subtransaction to recover to a consistent state, and less serious errors
<br>that don&#39;t have this requirement (e.g. invalid input to a data type<br>input function). If all the errors that we want to tolerate during a<br>bulk load fall into the latter category, we can do without<br>subtransactions.
<br></blockquote>
<div>&nbsp;</div>
<div>I think errors which occur after we have done a fast_heap_insert of the tuple generated from the current input row are the ones which would require the subtransaction to recover. Examples could be unique/primary key violation errors or FKey/triggers related errors. Any errors which occur before doing the heap_insert should&nbsp;not require any recovery according to me. 
</div>
<div>&nbsp;</div>
<div>The overhead of having a subtransaction per row is a very valid concern. But instead&nbsp;of using a per insert or a batch insert substraction, I am thinking&nbsp;that we can&nbsp;start off a subtraction and continue it till we encounter a failure. The moment an error is encountered, since we have the offending (already in heap) tuple around, we&nbsp;can call a simple_heap_delete on the same and commit (instead of aborting) this subtransaction after doing some minor cleanup. This current input data row can also be logged into a bad file. Recall that we need to only handle those errors in which the simple_heap_insert is successful, but the index insertion or the after row insert trigger causes an error. The rest of the load then can go ahead with the start of a new subtransaction. 
</div>
<div>&nbsp;</div>
<div>Regards,</div>
<div>Nikhils</div></div>-- <br>EnterpriseDB &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <a href="http://www.enterprisedb.com">http://www.enterprisedb.com</a> 

------=_Part_4228_10971707.1197703971110--