From: Daniel Verite <daniel@manitou-mail.org>
To: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Jim Nasby <Jim.Nasby@bluetreble.com>
Cc: Ronan Dunklau <ronan.dunklau@dalibo.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: pg_dump / copy bugs with "big lines" ?
Date: Wed, 02 Mar 2016 15:18:45 +0100
Message-ID: <10cb3300-67ab-4d3b-a930-7106d4dbac04@mm> (raw)
In-Reply-To: <149c79ee-9f41-48e1-83f8-3b2fe90af2b6@mm>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
I wrote:
> postgres=# \copy big2 to /dev/null
> lost synchronization with server: got message type "d", length -1568669676
>
> The backend aborts with the following backtrace:
>
> Program terminated with signal 6, Aborted.
> #0 0x00007f82ee68e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f82ee6913e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f82ee6c839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x00007f82ee6d1be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #4 0x00007f82ee6d698c in free () from /lib/x86_64-linux-gnu/libc.so.6
> #5 0x00000000007b5a89 in AllocSetDelete (context=0xffffffffffffffff)
> at aset.c:643
> #6 0x00000000007b63e3 in MemoryContextDelete (context=0x1fa58c8) at
> mcxt.c:229
> #7 0x000000000055fa25 in CopyTo (cstate=0x1fb1050) at copy.c:1967
The cause of the crash turns out to be, in enlargeStringInfo():
needed += str->len + 1; /* total space required now */
needed is an int and str->len is an int64, so it overflows when the
size has to grow beyond 2^31 bytes, fails to enlarge the buffer and
overwrites memory after it.
When fixing it with a local int64 copy of the variable, the backend
no longer crashes and COPY big2 TO 'file' appears to work.
However, getting it to the client with \copy big2 to 'file'
still produces the error in psql:
lost synchronization with server: got message type "d"
and leaves an empty file, so there are more problems to solve to
go beyond 2GB text per row.
Or maybe another approach would be to advertise that this is the maximum
for a row in text mode, and limit the backend's string buffer to this size
for the time being? Notwithstanding that there's still the other direction
client->server to test.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: daniel@manitou-mail.org, alvherre@2ndquadrant.com, robertmhaas@gmail.com, Jim.Nasby@bluetreble.com, ronan.dunklau@dalibo.com
Subject: Re: pg_dump / copy bugs with "big lines" ?
In-Reply-To: <10cb3300-67ab-4d3b-a930-7106d4dbac04@mm>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox