Received: from localhost (unknown [200.46.204.183]) by developer.postgresql.org (Postfix) with ESMTP id 456C02E002C for ; Tue, 22 Apr 2008 18:05:40 -0300 (ADT) Received: from developer.postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 19221-03 for ; Tue, 22 Apr 2008 18:05:37 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.226]) by developer.postgresql.org (Postfix) with ESMTP id 46BD82E0043 for ; Tue, 22 Apr 2008 18:05:37 -0300 (ADT) Received: by wr-out-0506.google.com with SMTP id 76so1410096wra.10 for ; Tue, 22 Apr 2008 14:05:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=kDcRoTyJfqIGA8kBmttROS9BLzu+PWeBTov7nZXNkV0=; b=m1s+nt0qWvXKhszoJgfI2LM+rIa0pLhREcn8CNommKJ4KBQdCaIzKw3OthKhc/bPXGc/Zf+400WC6gF+ZgKRkN4XzxyIrWfA7fGI4lUVlaaUt8dXWYaIDAmYjOXAgyVae+aREi85OFwOfCdtHBhH9wBc1/FOIRsv9nQSXynGnjo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=rl9G2isF9SqT5fLcHRpktgp15U0TTcDcAIcFDMPMRZMSNgoqm9VmttzVOuITW+MyXEeBK8u7zgQrwXE9thjegIl3Y9RmGLCFA9XvI0vF5woj23nNBpLQUt3mwp5EwiB+xpLyAzAUd2AAUrAgGOfSgT7/y7mRuOreLZrCzTqrN8g= Received: by 10.140.132.8 with SMTP id f8mr139047rvd.122.1208898335258; Tue, 22 Apr 2008 14:05:35 -0700 (PDT) Received: by 10.141.179.19 with HTTP; Tue, 22 Apr 2008 14:05:35 -0700 (PDT) Message-ID: <8a5d3c890804221405m69eb84b0k76f8e08b7f41952c@mail.gmail.com> Date: Tue, 22 Apr 2008 18:05:35 -0300 From: "Adonias Malosso" To: pgsql-performance@postgresql.org Subject: [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_20896_8119978.1208898335253" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 200804/332 X-Sequence-Number: 29825 ------=_Part_20896_8119978.1208898335253 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi all, I=B4m running a copy for a 37G CSV and receiving the following error: "invalid string enlargement request size 65536" The file has about 70 million lines with 101 columns, all them varchar. When I run the command with the whole file i receive the error after loadin= g about 29million lines. So i=B4ve spllited the file in 10 million lines with split: split --lines=3D10000000 And running the copy i receive the error on the 5th file: psql:/srv/www/htdocs/import/script_q2.sql:122: ERROR: invalid string enlargement request size 65536 CONTEXT: COPY temp_q2, line 3509639: ""000000009367276";"4";"DANIEL DO CARMO BARROS";"31-Jan-1986";"M";"1";"10";"3162906";"GILSON TEIXEIRA..." Any clues? My postgresql version is 8.2.4 the server is running suse linux with 1.5GB Sensitive changes in postgresql.conf are: shared_buffers =3D 512MB temp_buffers =3D 256MB checkpoint_segments =3D 60 I=B4d also like to know if there=B4s any way to optimize huge data load in operations like these. Regards Adonias Malosso ------=_Part_20896_8119978.1208898335253 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi all,

I=B4m running a copy for a 37G CSV and receiving the followi= ng error:

"invalid string enlargement request size 65536"<= br>
The file has about 70 million lines with 101 columns, all them varch= ar.

When I run the command with the whole file i receive the error after lo= ading
about 29million lines. So i=B4ve spllited the file in 10 million l= ines with split:

split --lines=3D10000000

And running the cop= y i receive the error on the 5th file:

psql:/srv/www/htdocs/import/script_q2.sql:122: ERROR:  invalid str= ing enlargement request size 65536
CONTEXT:  COPY temp_q2, line 350= 9639: ""000000009367276";"4";"DANIEL DO CARMO= BARROS";"31-Jan-1986";"M";"1";"10&= quot;;"3162906";"GILSON TEIXEIRA..."

Any clues?

My postgresql version is 8.2.4 the server is running = suse linux with 1.5GB
Sensitive changes in postgresql.conf are:

= shared_buffers =3D 512MB
temp_buffers =3D 256MB
checkpoint_segments = =3D 60

I=B4d also like to know if there=B4s any way to optimize huge data load= in operations like these.

Regards

Adonias Malosso
------=_Part_20896_8119978.1208898335253--