Subject: Re: pg_dump / copy bugs with "big lines" ?
To: Tom Lane <tgl@sss.pgh.pa.us>
References: <10cb3300-67ab-4d3b-a930-7106d4dbac04@mm>
 <56D70458.2010006@2ndquadrant.com> <21331.1456932193@sss.pgh.pa.us>
Cc: pgsql-hackers@postgresql.org
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <56D70703.9000007@2ndquadrant.com>
Date: Wed, 2 Mar 2016 16:30:11 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <21331.1456932193@sss.pgh.pa.us>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org


On 03/02/2016 04:23 PM, Tom Lane wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>> On 03/02/2016 03:18 PM, Daniel Verite wrote:
>>> However, getting it to the client with \copy big2 to 'file'
>>> still produces the error in psql:
>>> lost synchronization with server: got message type "d"
>>> and leaves an empty file, so there are more problems to solve to
>>> go beyond 2GB text per row.
>
>> My guess is this is a problem at the protocol level - the 'd' message is
>> CopyData, and all the messages use int32 to define length. So if there's
>> a 2GB row, it's likely to overflow.
>
> I'm too lazy to check the exact wording, but I don't think there's a hard
> and fast promise in the protocol doc that one CopyData message == one row.
> So we could probably subdivide a very wide line into multiple messages.

Well, actually we claim this [1]:

     Data that forms part of a COPY data stream. Messages sent from the
     backend will always correspond to single data rows, but messages
     sent by frontends might divide the data stream arbitrarily.

So that suggests 1:1 messages to rows, although I'm not sure how 
difficult would it be to relax this (or how much the clients might rely 
on this).

[1] http://www.postgresql.org/docs/9.5/static/protocol-message-formats.html

regards

--
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers