pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Ron Johnson <ron.l.johnson@cox.net>
To: PgSQL Performance ML <pgsql-performance@postgresql.org>
Subject: Re: LOCK TABLE & speeding up mass data loads
Date: 27 Jan 2003 03:08:20 -0600
Message-ID: <1043658500.818.398.camel@haggis> (raw)
In-Reply-To: <Pine.NEB.4.51.0301270808260.596@angelic.cynic.net>
References: <1043521053.818.32.camel@haggis>
	<1043521665.58142.135.camel@jester>
	<1043522349.818.36.camel@haggis>
	<Pine.NEB.4.51.0301270808260.596@angelic.cynic.net>

On Sun, 2003-01-26 at 17:10, Curt Sampson wrote:
> On Sun, 25 Jan 2003, Ron Johnson wrote:
> 
> > > Oh, and you're using COPY right?
> >
> > No.  Too much data manipulation to do 1st.  Also, by committing every
> > X thousand rows, then if the process must be aborted, then there's
> > no huge rollback, and the script can then skip to the last comitted
> > row and pick up from there.
> 
> I don't see how the amount of data manipulation makes a difference.
> Where you now issue a BEGIN, issue a COPY instead. Where you now INSERT,
> just print the data for the columns, separated by tabs. Where you now
> issue a COMMIT, end the copy.

Yes, create an input file for COPY.  Great idea. 

However, If I understand you correctly, then if I want to be able
to not have to roll-back and re-run and complete COPY (which may
entail millions of rows), then I'd have to have thousands of seperate
input files (which would get processed sequentially).

Here's what I'd like to see:
COPY table [ ( column [, ...] ) ]
    FROM { 'filename' | stdin }
    [ [ WITH ] 
          [ BINARY ] 
          [ OIDS ]
          [ DELIMITER [ AS ] 'delimiter' ]
          [ NULL [ AS ] 'null string' ] ]
    [COMMIT EVERY ... ROWS WITH LOGGING]  <<<<<<<<<<<<<
    [SKIP ... ROWS]          <<<<<<<<<<<<<

This way, if I'm loading 25M rows, I can have it commit every, say,
1000 rows, and if it pukes 1/2 way thru, then when I restart the 
COPY, it can SKIP past what's already been loaded, and proceed apace.

-- 
+---------------------------------------------------------------+
| Ron Johnson, Jr.        mailto:ron.l.johnson@cox.net          |
| Jefferson, LA  USA      http://members.cox.net/ron.l.johnson  |
|                                                               |
| "Fear the Penguin!!"                                          |
+---------------------------------------------------------------+




view thread (10+ messages)  latest in thread

Message-ID: <1043658500.818.398.camel@haggis>
Permalink:  ../1043658500.818.398.camel@haggis/
Also on:    postgresql.org/message-id/1043658500.818.398.camel@haggis

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: ron.l.johnson@cox.net
  Subject: Re: LOCK TABLE & speeding up mass data loads
  In-Reply-To: <1043658500.818.398.camel@haggis>
  References: <1043521053.818.32.camel@haggis> <1043521665.58142.135.camel@jester> <1043522349.818.36.camel@haggis> <Pine.NEB.4.51.0301270808260.596@angelic.cynic.net> <1043658500.818.398.camel@haggis>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox