Content-Type: text/plain; charset="iso-8859-15"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: pg_dump / copy bugs with "big lines" ?
From: "Daniel Verite" <daniel@manitou-mail.org>
To: "Alvaro Herrera" <alvherre@2ndquadrant.com>,
 "Robert Haas" <robertmhaas@gmail.com>,"Jim Nasby"
 <Jim.Nasby@bluetreble.com>,"Ronan Dunklau"
 <ronan.dunklau@dalibo.com>,"pgsql-hackers" <pgsql-hackers@postgresql.org>
In-Reply-To: <4023faba-72cc-4949-b11c-8c67a0ecf407@mm>
Date: Tue, 01 Mar 2016 22:31:19 +0100
Message-ID: <149c79ee-9f41-48e1-83f8-3b2fe90af2b6@mm>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

	I wrote:

> If splitting the table into 3 fields, each smaller than 512MB:
>=20
> postgres=3D# create table big2 as select
>  substring(binarycol from 1 for 300*1024*1024) as b1,
>  substring(binarycol from 1+300*1024*1024 for 300*1024*1024) as b2 ,
>  substring(binarycol from 1+600*1024*1024 for 400*1024*1024) as b3
> from big;

I've tried adding another large field to see what happens if the whole row
exceeds 2GB, and data goes to the client rather than to a file.

postgres=3D# alter table big2 add b4 bytea;
postgres=3D# update big2 set b4=3Db1;

So big2 has 4 bytea columns with 300+300+400+300 MB on a single row.

Then I'm trying to \copy this from a 9.5.1 backend with patch applied,
configured with --enable-debug, on Debian8 x86-64 with 64GB of RAM
and all defaults in postgresql.conf

My idea was to check if the client side was OK with that much data on
a single COPY row, but it turns out that the server is not OK anyway.

postgres=3D# \copy big2 to /dev/null
lost synchronization with server: got message type "d", length -1568669676

The backend aborts with the following backtrace:

Program terminated with signal 6, Aborted.
#0  0x00007f82ee68e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f82ee6913e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f82ee6c839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f82ee6d1be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f82ee6d698c in free () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00000000007b5a89 in AllocSetDelete (context=3D0xffffffffffffffff)
    at aset.c:643
#6  0x00000000007b63e3 in MemoryContextDelete (context=3D0x1fa58c8) at
mcxt.c:229
#7  0x000000000055fa25 in CopyTo (cstate=3D0x1fb1050) at copy.c:1967
#8  DoCopyTo (cstate=3Dcstate@entry=3D0x1fb1050) at copy.c:1778
#9  0x0000000000562ea9 in DoCopy (stmt=3Dstmt@entry=3D0x1f796d0,
    queryString=3DqueryString@entry=3D0x1f78c60 "COPY  big2 TO STDOUT ",
    processed=3Dprocessed@entry=3D0x7fff22103338) at copy.c:961
#10 0x00000000006b4440 in standard_ProcessUtility (parsetree=3D0x1f796d0,
    queryString=3D0x1f78c60 "COPY  big2 TO STDOUT ", context=3D<optimized o=
ut>,
    params=3D0x0, dest=3D0x1f79a30, completionTag=3D0x7fff22103680 "")
    at utility.c:541
#11 0x00000000006b1937 in PortalRunUtility (portal=3D0x1f26680,
    utilityStmt=3D0x1f796d0, isTopLevel=3D1 '\001', dest=3D0x1f79a30,
    completionTag=3D0x7fff22103680 "") at pquery.c:1183
#12 0x00000000006b2535 in PortalRunMulti (portal=3Dportal@entry=3D0x1f26680,
    isTopLevel=3DisTopLevel@entry=3D1 '\001', dest=3Ddest@entry=3D0x1f79a30,
    altdest=3Daltdest@entry=3D0x1f79a30,
    completionTag=3DcompletionTag@entry=3D0x7fff22103680 "") at pquery.c:13=
14
#13 0x00000000006b3022 in PortalRun (portal=3Dportal@entry=3D0x1f26680,
    count=3Dcount@entry=3D9223372036854775807,
    isTopLevel=3DisTopLevel@entry=3D1 '\001', dest=3Ddest@entry=3D0x1f79a30,
    altdest=3Daltdest@entry=3D0x1f79a30,
    completionTag=3DcompletionTag@entry=3D0x7fff22103680 "") at pquery.c:812
#14 0x00000000006b0393 in exec_simple_query (
    query_string=3D0x1f78c60 "COPY  big2 TO STDOUT ") at postgres.c:1104
#15 PostgresMain (argc=3D<optimized out>, argv=3Dargv@entry=3D0x1f0d240,
    dbname=3D0x1f0d0f0 "postgres", username=3D<optimized out>) at postgres.=
c:4030
#16 0x000000000047072b in BackendRun (port=3D0x1f2d230) at postmaster.c:4239
#17 BackendStartup (port=3D0x1f2d230) at postmaster.c:3913
#18 ServerLoop () at postmaster.c:1684
#19 0x000000000065b8cd in PostmasterMain (argc=3Dargc@entry=3D3,
    argv=3Dargv@entry=3D0x1f0c330) at postmaster.c:1292
#20 0x0000000000471161 in main (argc=3D3, argv=3D0x1f0c330) at main.c:223


The server log has this:

STATEMENT:  COPY  big2 TO STDOUT
*** glibc detected *** postgres: postgres postgres [local] COPY: double free
or corruption (out): 0x00007f8234929010 ***
=3D=3D=3D=3D=3D=3D=3D Backtrace: =3D=3D=3D=3D=3D=3D=3D=3D=3D
/lib/x86_64-linux-gnu/libc.so.6(+0x75be6)[0x7f82ee6d1be6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f82ee6d698c]
postgres: postgres postgres [local] COPY[0x7b5a89]
postgres: postgres postgres [local] COPY(MemoryContextDelete+0x43)[0x7b63e3]
postgres: postgres postgres [local] COPY[0x55fa25]
postgres: postgres postgres [local] COPY(DoCopy+0x479)[0x562ea9]
postgres: postgres postgres [local]
COPY(standard_ProcessUtility+0x590)[0x6b4440]
postgres: postgres postgres [local] COPY[0x6b1937]
postgres: postgres postgres [local] COPY[0x6b2535]
postgres: postgres postgres [local] COPY(PortalRun+0x202)[0x6b3022]
postgres: postgres postgres [local] COPY(PostgresMain+0x1493)[0x6b0393]
postgres: postgres postgres [local] COPY[0x47072b]
postgres: postgres postgres [local] COPY(PostmasterMain+0xe7d)[0x65b8cd]
postgres: postgres postgres [local] COPY(main+0x3e1)[0x471161]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f82ee67aead]
postgres: postgres postgres [local] COPY[0x4711c9]


I will try other tests without bytea exported in text format but with
several huge text columns.

Best regards,
--=20
Daniel V=E9rit=E9
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite


--=20
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers