Content-Type: text/plain; charset="iso-8859-15"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: pg_dump / copy bugs with "big lines" ?
From: "Daniel Verite" <daniel@manitou-mail.org>
To: "Tom Lane" <tgl@sss.pgh.pa.us>
CC: "Alvaro Herrera" <alvherre@2ndquadrant.com>,"Robert Haas"
 <robertmhaas@gmail.com>,"Jim Nasby" <Jim.Nasby@bluetreble.com>,"Ronan
 Dunklau" <ronan.dunklau@dalibo.com>,"pgsql-hackers"
 <pgsql-hackers@postgresql.org>
In-Reply-To: <27428.1456868923@sss.pgh.pa.us>
Date: Thu, 03 Mar 2016 17:19:55 +0100
Message-ID: <7c0836cc-ae43-445e-9c83-ab569e2525df@mm>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

	Tom Lane wrote:

> BTW, is anyone checking the other side of this, ie "COPY IN" with equally
> wide rows?  There doesn't seem to be a lot of value in supporting dump
> if you can't reload ...

Indeed, the lines bigger than 1GB can't be reloaded :(

My test: with a stock 9.5.1 plus Alvaro's patch with my additional
int64 fix mentioned upthread, creating this table and filling
all columns with 200MB of text each:

create table bigtext(t1 text ,t2 text, t3 text, t4 text,
					 t5 text, t6 text, t7 text);

# \copy bigtext to '/var/tmp/bigtext.sql'

That part does work as expected, producing this huge single line:
$ wc /var/tmp/bigtext.sql
	 1	    7 1468006407 /var/tmp/bigtext.sql

But reloading it fails:

# create table bigtext2 (like bigtext);
CREATE TABLE

# copy bigtext2 from '/var/tmp/bigtext.sql';
ERROR:	54000: out of memory
DETAIL:  Cannot enlarge string buffer containing 1073676288 bytes by 65536
more bytes.
CONTEXT:  COPY bigtext2, line 1
LOCATION:  enlargeStringInfo, stringinfo.c:278


# \copy bigtext2 from '/var/tmp/bigtext.sql'
ERROR:	54000: out of memory
DETAIL:  Cannot enlarge string buffer containing 1073741808 bytes by 8191
more bytes.
CONTEXT:  COPY bigtext2, line 1
LOCATION:  enlargeStringInfo, stringinfo.c:278


Best regards,
--=20
Daniel V=E9rit=E9
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite


--=20
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers