pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Ron Johnson <ron.l.johnson@cox.net>
To: PgSQL Performance ML <pgsql-performance@postgresql.org>
Subject: Re: LOCK TABLE & speeding up mass data loads
Date: 27 Jan 2003 03:54:25 -0600
Message-ID: <1043661264.9231.8.camel@haggis> (raw)
In-Reply-To: <3E354CFB.32732.A38120C@localhost>
References: <Pine.NEB.4.51.0301270808260.596@angelic.cynic.net>
	<3E354CFB.32732.A38120C@localhost>

On Mon, 2003-01-27 at 03:45, Shridhar Daithankar wrote:
> On 27 Jan 2003 at 3:08, Ron Johnson wrote:
> 
> > Here's what I'd like to see:
> > COPY table [ ( column [, ...] ) ]
> >     FROM { 'filename' | stdin }
> >     [ [ WITH ] 
> >           [ BINARY ] 
> >           [ OIDS ]
> >           [ DELIMITER [ AS ] 'delimiter' ]
> >           [ NULL [ AS ] 'null string' ] ]
> >     [COMMIT EVERY ... ROWS WITH LOGGING]  <<<<<<<<<<<<<
> >     [SKIP ... ROWS]          <<<<<<<<<<<<<
> > 
> > This way, if I'm loading 25M rows, I can have it commit every, say,
> > 1000 rows, and if it pukes 1/2 way thru, then when I restart the 
> > COPY, it can SKIP past what's already been loaded, and proceed apace.
> 
> IIRc, there is a hook to \copy, not the postgreSQL command copy for how many 

I'll have to look into that.

> transactions you would like to see. I remember to have benchmarked that and 
> concluded that doing copy in one transaction is the fastest way of doing it.

Boy Scout motto: Be prepared!!  (Serves me well as a DBA.)

So it takes a little longer.  In case of failure, the time would be
more than made up.  Also, wouldn't the WAL grow hugely if many millions
of rows were inserted in one txn?

-- 
+---------------------------------------------------------------+
| Ron Johnson, Jr.        mailto:ron.l.johnson@cox.net          |
| Jefferson, LA  USA      http://members.cox.net/ron.l.johnson  |
|                                                               |
| "Fear the Penguin!!"                                          |
+---------------------------------------------------------------+




view thread (10+ messages)  latest in thread

Message-ID: <1043661264.9231.8.camel@haggis>
Permalink:  ../1043661264.9231.8.camel@haggis/
Also on:    postgresql.org/message-id/1043661264.9231.8.camel@haggis

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: ron.l.johnson@cox.net
  Subject: Re: LOCK TABLE & speeding up mass data loads
  In-Reply-To: <1043661264.9231.8.camel@haggis>
  References: <Pine.NEB.4.51.0301270808260.596@angelic.cynic.net> <3E354CFB.32732.A38120C@localhost> <1043661264.9231.8.camel@haggis>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox