public inbox for [email protected]  

help / color / mirror / Atom feed

From: Josh Berkus <[email protected]>
To: [email protected]
Subject: Re: VLDB Features
Date: Thu, 20 Dec 2007 12:04:26 -0800
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<1197717031.7974.23.camel@hannu-laptop>
	<[email protected]>

Tom,

> Sure ... but you'll find that it's not large enough to be useful.
> Once you remove all the interesting consistency checks such as
> unique indexes and foreign keys, the COPY will tend to go through
> just fine, and then you're still stuck trying to weed out bad data
> without very good tools for it.  The only errors we could really
> separate out without subtransaction fencing are extremely trivial
> ones like too many or too few fields on a line ... which can be
> caught with a sed script.

Speaking as someone who did a LOT of DW load design only a couple years ago, 
I'll say that the "special case" of no triggers, no constraint checks except 
length, and type-safety check actually constitutes about 50% of DW bulk 
loading.  The only exception to that is unique indexes, which would normally 
be included and would be the difficult thing.

Also, "special case bulk loading" would in fact give users of other types of 
applications a lot more flexibility -- they could always load into a holding 
table just to clean up the type safety issues and then merge into the real 
table.  

So I don't agree that the "load into new partition without dependancies" is 
too much of a special case to be worth pursuing.  It might be a bad idea for 
other reasons, but not because it's too obscure.

--Josh

view thread (39+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected]
  Subject: Re: VLDB Features
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox