Subject: Re: pg_dump / copy bugs with "big lines" ?
To: pgsql-hackers@postgresql.org
References: <10cb3300-67ab-4d3b-a930-7106d4dbac04@mm>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <56D70458.2010006@2ndquadrant.com>
Date: Wed, 2 Mar 2016 16:18:48 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <10cb3300-67ab-4d3b-a930-7106d4dbac04@mm>
Content-Type: text/plain; charset=iso-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

Hi,

On 03/02/2016 03:18 PM, Daniel Verite wrote:
> 	I wrote:
>
>> postgres=# \copy big2 to /dev/null
>> lost synchronization with server: got message type "d", length -1568669676
>>
>> The backend aborts with the following backtrace:
>>
>> Program terminated with signal 6, Aborted.
>> #0  0x00007f82ee68e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x00007f82ee6913e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2  0x00007f82ee6c839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3  0x00007f82ee6d1be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x00007f82ee6d698c in free () from /lib/x86_64-linux-gnu/libc.so.6
>> #5  0x00000000007b5a89 in AllocSetDelete (context=0xffffffffffffffff)
>>     at aset.c:643
>> #6  0x00000000007b63e3 in MemoryContextDelete (context=0x1fa58c8) at
>> mcxt.c:229
>> #7  0x000000000055fa25 in CopyTo (cstate=0x1fb1050) at copy.c:1967
>
> The cause of the crash turns out to be, in enlargeStringInfo():
>
> needed += str->len + 1; 	/* total space required now */
>
> needed is an int and str->len is an int64, so it overflows when the
> size has to grow beyond 2^31 bytes, fails to enlarge the buffer and
> overwrites memory after it.
>
> When fixing it with a local int64 copy of the variable, the backend
> no longer crashes and COPY big2 TO 'file' appears to work.
>
> However, getting it to the client with \copy big2 to 'file'
> still produces the error in psql:
>     lost synchronization with server: got message type "d"
> and leaves an empty file, so there are more problems to solve to
> go beyond 2GB text per row.

My guess is this is a problem at the protocol level - the 'd' message is 
CopyData, and all the messages use int32 to define length. So if there's 
a 2GB row, it's likely to overflow.

regards

--
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers