From: Adonias Malosso <malosso@gmail.com>
To: pgsql-performance@postgresql.org
Subject: [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536"
Date: Tue, 22 Apr 2008 18:05:35 -0300
Message-ID: <8a5d3c890804221405m69eb84b0k76f8e08b7f41952c@mail.gmail.com> (raw)
Hi all,
I´m running a copy for a 37G CSV and receiving the following error:
"invalid string enlargement request size 65536"
The file has about 70 million lines with 101 columns, all them varchar.
When I run the command with the whole file i receive the error after loading
about 29million lines. So i´ve spllited the file in 10 million lines with
split:
split --lines=10000000
And running the copy i receive the error on the 5th file:
psql:/srv/www/htdocs/import/script_q2.sql:122: ERROR: invalid string
enlargement request size 65536
CONTEXT: COPY temp_q2, line 3509639: ""000000009367276";"4";"DANIEL DO
CARMO BARROS";"31-Jan-1986";"M";"1";"10";"3162906";"GILSON TEIXEIRA..."
Any clues?
My postgresql version is 8.2.4 the server is running suse linux with 1.5GB
Sensitive changes in postgresql.conf are:
shared_buffers = 512MB
temp_buffers = 256MB
checkpoint_segments = 60
I´d also like to know if there´s any way to optimize huge data load in
operations like these.
Regards
Adonias Malosso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: malosso@gmail.com
Subject: Re: [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536"
In-Reply-To: <8a5d3c890804221405m69eb84b0k76f8e08b7f41952c@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox