From: Josh Berkus <josh@agliodbs.com>
Organization: PostgreSQL @ Sun
To: pgsql-hackers@postgresql.org
Subject: Re: VLDB Features
Date: Thu, 20 Dec 2007 12:04:26 -0800
User-Agent: KMail/1.9.7
References: <1197371508.4255.1327.camel@ebony.site>
	<1197717031.7974.23.camel@hannu-laptop>
	<26961.1197826056@sss.pgh.pa.us>
In-Reply-To: <26961.1197826056@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200712201204.26940.josh@agliodbs.com>

Tom,

> Sure ... but you'll find that it's not large enough to be useful.
> Once you remove all the interesting consistency checks such as
> unique indexes and foreign keys, the COPY will tend to go through
> just fine, and then you're still stuck trying to weed out bad data
> without very good tools for it.  The only errors we could really
> separate out without subtransaction fencing are extremely trivial
> ones like too many or too few fields on a line ... which can be
> caught with a sed script.

Speaking as someone who did a LOT of DW load design only a couple years ago, 
I'll say that the "special case" of no triggers, no constraint checks except 
length, and type-safety check actually constitutes about 50% of DW bulk 
loading.  The only exception to that is unique indexes, which would normally 
be included and would be the difficult thing.

Also, "special case bulk loading" would in fact give users of other types of 
applications a lot more flexibility -- they could always load into a holding 
table just to clean up the type safety issues and then merge into the real 
table.  

So I don't agree that the "load into new partition without dependancies" is 
too much of a special case to be worth pursuing.  It might be a bad idea for 
other reasons, but not because it's too obscure.

--Josh