public inbox for [email protected]  

help / color / mirror / Atom feed
From: =?ISO-8859-2?Q?Micha=B3_Zaborowski?= <[email protected]>
To: Tom Lane <[email protected]>
Cc: Neil Conway <[email protected]>
Cc: Hannu Krosing <[email protected]>
Cc: Josh Berkus <[email protected]>
Cc: [email protected]
Cc: Simon Riggs <[email protected]>
Subject: Re: VLDB Features
Date: Tue, 18 Dec 2007 16:01:52 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<1197717031.7974.23.camel@hannu-laptop>
	<[email protected]>

2007/12/16, Tom Lane <[email protected]>:
> Hannu Krosing <[email protected]> writes:
> > But can't we _define_ such a subset, where we can do a transactionless
> > load ?
>
> Sure ... but you'll find that it's not large enough to be useful.
> Once you remove all the interesting consistency checks such as
> unique indexes and foreign keys, the COPY will tend to go through
> just fine, and then you're still stuck trying to weed out bad data
> without very good tools for it.  The only errors we could really
> separate out without subtransaction fencing are extremely trivial
> ones like too many or too few fields on a line ... which can be
> caught with a sed script.
>
I have dump file. I would like to load it ASAP.
Constraints will be applied at the end, so any problem can be detected.
I would like it to be as direct as possible and as bulk as possibe - just
allocate pages and fill them with the data. Maybe it should be different
mode - single user or so. Right now I can save some IO - like turn off
fsync, but that is all :(

I got something like that:
http://www.tbray.org/ongoing/When/200x/2007/10/30/WF-Results
I have no idea how to load single file in many threads, but... the point is
that it can be much faster that single-thread load - surprisingly - at
least for me.

-- 
Regards,
  Michał Zaborowski (TeXXaS)


view thread (39+ messages)  latest in thread

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: VLDB Features
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox