pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Copy Command out of memory
3+ messages / 3 participants
[nested] [flat]

* Copy Command out of memory
@ 2005-12-14 23:18 Kevin Keith <kkeith@borderware.com>
  2005-12-15 05:24 ` Re: Copy Command out of memory Tom Lane <tgl@sss.pgh.pa.us>
  2005-12-15 21:54 ` Re: Copy Command out of memory Jim C. Nasby <jnasby@pervasive.com>
  0 siblings, 2 replies; 3+ messages in thread

From: Kevin Keith @ 2005-12-14 23:18 UTC (permalink / raw)
  To: pgsql-admin

I was trying to run a bulk data load using the COPY command on PGSQL 8.1.0.

After loading about 3,500,000 records it ran out of memory - I am 
assuming because it ran out of space to store such a large transaction. 
Does the COPY command offer a similar feature to Oracle's SQL*Loader 
where you can specify the number of records to load between commit 
statements, or will I have to break the file I am loading into smaller 
files?

Or can a transaction be bypassed altogether with the COPY command since 
any failure (the load is going to an empty table) could easily be solved 
with  a reload of the data anyway.

Thanks,

Kevin



^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: Copy Command out of memory
  2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
@ 2005-12-15 05:24 ` Tom Lane <tgl@sss.pgh.pa.us>
  1 sibling, 0 replies; 3+ messages in thread

From: Tom Lane @ 2005-12-15 05:24 UTC (permalink / raw)
  To: Kevin Keith <kkeith@borderware.com>; +Cc: pgsql-admin

Kevin Keith <kkeith@borderware.com> writes:
> I was trying to run a bulk data load using the COPY command on PGSQL 8.1.0.
> After loading about 3,500,000 records it ran out of memory - I am 
> assuming because it ran out of space to store such a large transaction. 

COPY per se shouldn't eat memory.

Perhaps you have some AFTER INSERT triggers on this table (note foreign
key constraints would create such)?  The pending-trigger queue could
have overflowed memory.  Offhand I can't think of another reason.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: Copy Command out of memory
  2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
@ 2005-12-15 21:54 ` Jim C. Nasby <jnasby@pervasive.com>
  1 sibling, 0 replies; 3+ messages in thread

From: Jim C. Nasby @ 2005-12-15 21:54 UTC (permalink / raw)
  To: Kevin Keith <kkeith@borderware.com>; +Cc: pgsql-admin

On Wed, Dec 14, 2005 at 06:18:07PM -0500, Kevin Keith wrote:
> Or can a transaction be bypassed altogether with the COPY command since 
> any failure (the load is going to an empty table) could easily be solved 
> with  a reload of the data anyway.

Work along those lines is currently in-progress. Search the -hackers
archive if you're curious.
-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461



^ permalink  raw  reply  [nested|flat] 3+ messages in thread


end of thread, other threads:[~2005-12-15 21:54 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
2005-12-15 05:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-12-15 21:54 ` Jim C. Nasby <jnasby@pervasive.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox