pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedCopy Command out of memory
3+ messages / 3 participants
[nested] [flat]
* Copy Command out of memory
@ 2005-12-14 23:18 Kevin Keith <kkeith@borderware.com>
2005-12-15 05:24 ` Re: Copy Command out of memory Tom Lane <tgl@sss.pgh.pa.us>
2005-12-15 21:54 ` Re: Copy Command out of memory Jim C. Nasby <jnasby@pervasive.com>
0 siblings, 2 replies; 3+ messages in thread
From: Kevin Keith @ 2005-12-14 23:18 UTC (permalink / raw)
To: pgsql-admin
I was trying to run a bulk data load using the COPY command on PGSQL 8.1.0.
After loading about 3,500,000 records it ran out of memory - I am
assuming because it ran out of space to store such a large transaction.
Does the COPY command offer a similar feature to Oracle's SQL*Loader
where you can specify the number of records to load between commit
statements, or will I have to break the file I am loading into smaller
files?
Or can a transaction be bypassed altogether with the COPY command since
any failure (the load is going to an empty table) could easily be solved
with a reload of the data anyway.
Thanks,
Kevin
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Copy Command out of memory
2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
@ 2005-12-15 05:24 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 3+ messages in thread
From: Tom Lane @ 2005-12-15 05:24 UTC (permalink / raw)
To: Kevin Keith <kkeith@borderware.com>; +Cc: pgsql-admin
Kevin Keith <kkeith@borderware.com> writes:
> I was trying to run a bulk data load using the COPY command on PGSQL 8.1.0.
> After loading about 3,500,000 records it ran out of memory - I am
> assuming because it ran out of space to store such a large transaction.
COPY per se shouldn't eat memory.
Perhaps you have some AFTER INSERT triggers on this table (note foreign
key constraints would create such)? The pending-trigger queue could
have overflowed memory. Offhand I can't think of another reason.
regards, tom lane
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Copy Command out of memory
2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
@ 2005-12-15 21:54 ` Jim C. Nasby <jnasby@pervasive.com>
1 sibling, 0 replies; 3+ messages in thread
From: Jim C. Nasby @ 2005-12-15 21:54 UTC (permalink / raw)
To: Kevin Keith <kkeith@borderware.com>; +Cc: pgsql-admin
On Wed, Dec 14, 2005 at 06:18:07PM -0500, Kevin Keith wrote:
> Or can a transaction be bypassed altogether with the COPY command since
> any failure (the load is going to an empty table) could easily be solved
> with a reload of the data anyway.
Work along those lines is currently in-progress. Search the -hackers
archive if you're curious.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2005-12-15 21:54 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-12-14 23:18 Copy Command out of memory Kevin Keith <kkeith@borderware.com>
2005-12-15 05:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-12-15 21:54 ` Jim C. Nasby <jnasby@pervasive.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox