Subject: Re: [DOCS] Let's document a bytea bug
To: Bruce Momjian <bruce@momjian.us>, "Andrey M. Borodin"
 <x4mmm@yandex-team.ru>
Cc: Anna Akenteva <a.akenteva@postgrespro.ru>, pgsql-docs@postgresql.org
References: <fe9b3722df94f7bdb08768f50ee8fe59@postgrespro.ru>
 <30380A0E-0188-4935-92B1-1F475C62C254@yandex-team.ru>
 <20200803202042.GC17519@momjian.us>
From: Joe Conway <mail@joeconway.com>
Autocrypt: addr=mail@joeconway.com; keydata=
 xsFNBEpXMCsBEADDnXUQzjlyi/cX02Gtdy2CLcroE5CsC7DJKdOBDbfgn0kfiIYoV5JniG4l
 VyzZUodY8yUAagqLYolh0UkBzs9N+qkm7erde4ypw3jzVQ37BuzIvk3nMUbuDZDgxWqX+nVS
 sKc+BQ5BpzgCHg48leoRO2ohjvYnUhgH3j2rFZCzaj6qQ7mv+XoxOJmUlVQtG06Jwkk7Vu14
 7U9nMMM6hyUKzVnmCphnlcMNo26UyVU70MwFfFJgcI0c5fpp8byN56eD6VJVnufO5WAuEhzE
 qcrSJR2FAlmM90GBY+6vP29twLDCHuSFvrnujNCx/BvCC/a3/gPvyAFp4JtMm9eXAmq3m/Kw
 94nTJXVdcbQeQQDp3KIG7MmWS4lnGvPn8v0CjgNaLvZXFLo1FgmUVsyEq1Lww4iRLa6sbpXJ
 ESx15UEue1k1YZM9C+4F/o3aeKNsAienjw2EXFzcaxIg/C4P493VMi3Qa8ycVxR5iYhUbYdo
 DFIUQhbFNsYfrtW/qZAELT3FCYFpZYG01e9Hj+cBrXXgyDDkQ5Lq4mlvmkRvuxn61V6Au4HA
 0sJiCox5pM1FvzT+aI8HY1BYaiB9Pl4fhpKgmhhlSuglk9v39S4jmlUIb45iLAUVpeNM6Qjm
 69pf5da9sm4aGFa7YlDSKf/WcU7z9ITZxsilOi2n7YJiwG7kTQARAQABzR9Kb2UgQ29ud2F5
 IDxtYWlsQGpvZWNvbndheS5jb20+wsF8BBMBAgAmAhsDBwsJCAcDAgEGFQgCCQoLBBYCAwEC
 HgECF4AFAlpoksUCGQEACgkQMyt+aLaZQ0p4XQ/9F4WSf4ZFuL3ae356ApVqnLxphHXeXHli
 ePZe7qosvXMQLN9Y2m5cQwqojsQh4pZujIi4Iwxw7ORg6HbMO8dMhyu8TlDavErcgO561hG7
 /TCcCLf5SkadxBqYWAJH09W8S2iC3gQtS+qs4Y7sIvRi881EsPzG5Lrzu7Ei8ufqsSnbLUQk
 zAaKjK3IzAFvzA3VCg/8Aqhf8obhqL0j5I5DNOpAt2U4gkEzIrX1g5xSXpRLMHLEK7hPViY8
 pFQyRvyZioE4hckNsnidCgiv4HefJUFRIVxDkHBeLhTzqTPloesFT/bskhckIS6F4suMx044
 WTTW6UiARuWw3aaBTmxay2rjPCsGaJPawzY0O3HlNlZherCzuBpa+Qk1kMLkcxMlYjCrMgIR
 YyBpIK46ltPQaog4Ujh21w1Z8m/ufG5v/HTuLdXM9hGfTlW7aY6psCAV/MwfycDHLRMc+UY7
 W2cp6DlWQ7GHLP+vzvBcl5Odi04EhcYnHMZjKvRFsxhE0LZcTejrtxPkDgR9G59irlIx1+py
 ziK2B0hs7Er1Yq7qQGQiQGLoglSNDsvsUeyMGyabasmUd5xrRO35SpGv+hyYGAZ3Pyk9Vtxl
 CO7BJkZiV9YSMHp6hOdmYZd4SZRChd9sKd6eSw1xIlBALz1ZHuQVMBuZvuVGFsS81Zqckb/2
 D5rOwU0ESlcx7QEQAKhm5UhGfbe3hM4yCaa/pPaNaswlB2tK1DHAjDlDm7gDVEodfsrvAFUU
 unLodlvCd0g/GpLmXXdROU/I+HusVqQhDbVeHMwO2VUVHamRWqDfrCjMYVxLz80d96TYyeyW
 6RUNkJmXJAOKgy8veI7Z1F+j1gXQR7UlzonnSkWbs0oQ3RglvyqhkUzWtXU7F+j2hQ2vQsct
 Lnq3lYo0bYkZ3K6RyKZXSly9Hd7DzYkMzz7StNRlyeQJxplPrVPup7N8SqDrI126sq0cwdUB
 Gn1sKVoO+bGidBjhJgxouNPkMJlPjbKOgb7tv5ZnM33FGXNdC5sPKoZ4mC+gaqRHYaH/V61v
 fCo1CYNLIK+Cj+Oua8mQ6CPeOb5VKcJ0JM96eGNlzF9EawppOG7xCDcbmBEr0m1go9MHEUel
 0L1i8RSDYOHQQkLT2MVWYf4UsRlEGbME+kF7DEX9Fvie7qglFTXC2d+YVN73AnXcoM/IYUOR
 DXjVVaqQoHuA4uxwJ31o/rEaOihr2yCScr8mB8J3b3KyNfuCBm5gN0tUQw+t5szMYurDpZ1G
 L4O7btaQ5u4iJMfRTCnPrzW+wkdVrao7AmRNWEtSlXsycClKFCHz/Ej5rpXOKSp+a/xcaFX1
 v2eS/+Ujp5ESu9p57JwlP6aSqYBiOALRJXZ9DVtZL96okjzXjrd7ABEBAAHCw34EGAEIAAkF
 AkpXMe0CGwICKQkQMyt+aLaZQ0rBXSAEGQEIAAYFAkpXMe0ACgkQN/L3QzX32GV1Jw//fhvL
 Mk1YNS/2N5jc5HYmEjcL2OWBLLrO5RMCaOgCNmYsepkigy2U+GSloVy8Y40/UfYzIoNf1DiB
 Qyj5Ipa4eMjfKA7+1q43a5wkDDWogFNflKTTNemwrHDKR1WWDrxt8uAgJdlBGDQA7A0q3D2i
 Xhd7ALLOHhuXDY1++N9TK9KMfV90Oay0mIMLQhTG/UgsgrYYb78H2pk8XMYZY29vgZlj9VH2
 f1RpOoh7lCaB+2glN27Rzh17q0/zEjwXk4sDyYHRIycaqnM4ozSnliXyoWM5bvmqSGBCm5jl
 BjNW+cFbFy15U595AIXXiUXl8hsTF2JWwgHr+hYdtoEH3oEOxzM18ENXKTTgN+BafdMsah4e
 HIcVvrVrH6rlsPwCvk3o/PJ07qVbUW+dbZG8UB1JwqUUqGhrCk87kztajlvZGZWxWDSB4Lnb
 T6Jc0L7/YXG8U3sJ2OtZ95Lal5XfNo0yBdPflmalH3xf/EcxaYFETbDrY5UgEVa2HBw8XzKk
 rTbM8rD318M+jBOuL6o3cGMOeFOm4C51fJZmOCFd03LhMkoj9IbLIPGKosIBsxb7zP5YLAbI
 b3el6pvo9CKtNuR+IcVOoFtH+KVOxGsbbxFEFbsAmPMreF1hIv+l+HXBaDhUkaUmpJ1Mhxol
 aM6O/LGG4VQV7wBQFunVaN8EPqMqYDph3A//TjQVd8fqvw5Ezj1958IUmV0jxeGX2m/fspGh
 njF+6Adx3PFbZoKyW71cA/TnlHVYCaizjYYRNMC3g5VFytOYOZFi0GKsxQTX5PcPhDjFFTUv
 FVVZwDYUK3QlC+wGDTeNWfAJIZnTPPIXSiHzh3si+ngKjE+ngcRF1VY5K4xvc6ss6uq8kI8L
 Ct+TpOMr99ijaYEbMZ8WJagynD5BLNYxpP6CCTMDi1Xqydg0E/mhR9w9aUOvdTZw1OOKtLnz
 B1E0wFbzJXEvjP2eEVNlStS5RTawFDpqxGwXJcHqe8qtg8B+C1r47RmHnKKm+DTOPFQWTj83
 l775aNMzAHm+Pr1TdSA5XRyQ3wxwYKfhp1suKhILUZLjbiCQf92RZQ6yEdXskMr2A4n0GQWq
 Q8k7r9QPqtUP7lsiyyDem7r9h5mAm94r3Td22L3JaSHkBD+4jC4gIXeYoyd+lTz/zYOx0Fri
 J8dXHalmyTGsSxV7FXv1/thVEf6R7TDdVwhqf/WL3uWTmVBR0tJscG+9iCVch/OG/uCDkWZQ
 sOTotqj6f5jSZI+NOUqt62+8i8gjYEa7zRZ2HXFOTV8UduCjheWB2VCUj5EjFjTMjfTVkyZ5
 SiFht7bM5l/odoYU6vn37/5s/ZsMTZX1zG5oXv6iULrHLYHhUxR93FlVyn4DiHsWND/Idj7O
 wU0ESlcyJwEQAOkTBb9yDhJbMUgvhM11rZwT5tm4Y9TqtEHn0Zy3t9g7bdFFpMvav/KENd3o
 AtLFpMDf+H3AggFk4ftUwJwiVgJ88ilvCynJUGXiuYIaexY4DLgn4xpnuiEpYEFVdWnlw7dW
 VTc62exfqIz9bSWRzwfBCY9ruYGEb4RDPDSNSAVyI7sxHzef2asiYxIcxrTrw5VugWNlPZcV
 5/EJ6PUvATjBF2TBkXV7KOciQng2tsQGrGMkY5mduNqwpuh6zfPcVF8LeObe96wv5ZhPRpO7
 9nef7hnK2lJogp3JIo558Jlbz9WHtQEMZR85+bUhtI825QyNAFz3Jrn7NMgvDikc2OrWo7YM
 gMC5hDSWVFqA6/EQCNnDWGABWgeYHZFpnPwsvUWIYdhSilUuj/Tuzvz9ZmucFNbQbauDQw6V
 Q38ofGnoYDZFJsGncprB8dBi4tDrIQ+1RlIh6C2Z/eMipqJOT26+spluTjouvnKT0S5yOgyX
 0PjbsysgwQdCGNJLHOjhHbSpSmOLaduV3CQo/0+DHT/TBjYfIXjTWouY9TkGxG4eNrxU0u2x
 Ay5bMqOPmsFdjLTWlQUlF/fTMhB54XwI3FHWgnSnXZzStDTmTebLNdT/ftgliAzA81uMj49j
 0exv731/v+7udLA1bV8gnZ01zQCASDpWiRQR3fgwcugSUqgRABEBAAHCwV8EGAEIAAkFAkpX
 MicCGwwACgkQMyt+aLaZQ0pwAQ//bjcWnZg/jjRQ9gbZUGMqniItZYRglBMKIqt4Fia379Jm
 HwTvavnFkJ8XMZ56UB0FIrgS+sUkRH6cPRQR+7Qi392LD021DXgSsz9CwFHjFyBGHwLEOTRc
 fYQbtJy0shHDJB4aQTOX3ERDH1PsvJNuevmQMzS0DWFav9+xMz9rKP4N+HffoBIZE0C1xIE4
 3nD4eLsbycte9sVIrmlNuUti3qUxJAQw8HwfJ6ZbBInHxquApR16uD1u99o6XlndFrDlY22t
 RmHCM0bR81GfGNdcU3Uo+rG/R/k4qa7s9/dgKvMbyH3fHhp/ceKag80Xo8IFurRl0ZJP3sHJ
 2QDHCVLat7jRZ+43hi1WlIhFbrgn6IyI0i7XR/W8JjrC5MsKq4TUwGH077sU/kcHYebVJZRb
 UUst2hAGHDFVBcG12qoKf+ltL9qXJc1y7BGeCoUW6QjOpljpq6ZL4FQUsM0RSRjs5egE3szP
 cIf5SyPK6WDOApoAq6M7BBFMGDZwEylYMtr0YekA1u86UA9D2xwLHEbBBp/uiby1c9JbPJ1P
 n8zJP8WZNeRw4Q9TtqVK09+oLirMUSpIDd6KdZ1VgRxOK2re7tjDvkVuYsSrsiJ+1iJNEnp9
 iK0ok0DlJpSCe6KhkxpaTdeoWMXdKuJWec0NIqoAd54ZgBPnr+UPxTixgPq/p6Q=
Message-ID: <8ba460ba-9947-73e0-6920-a8a2bca436b9@joeconway.com>
Date: Mon, 3 Aug 2020 16:44:38 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <20200803202042.GC17519@momjian.us>
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature";
 boundary="EnhV0MYJhJSiTFsOr5HVOtDpWFJSsYG4G"
Precedence: bulk

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--EnhV0MYJhJSiTFsOr5HVOtDpWFJSsYG4G
Content-Type: multipart/mixed; boundary="dpmTzZF8qHq58NHEuWMhn64KWUzWG6nBn"

--dpmTzZF8qHq58NHEuWMhn64KWUzWG6nBn
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

On 8/3/20 4:20 PM, Bruce Momjian wrote:
> On Fri, Jul 31, 2020 at 10:13:48AM +0500, Andrey M. Borodin wrote:
>> Hi Anna!
>>=20
>> > 23 =D0=BC=D0=B0=D1=8F 2018 =D0=B3., =D0=B2 20:33, Anna Akenteva <a.a=
kenteva@postgrespro.ru> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0=
):
>> >=20
>> >=20
>> > Some time ago I've encountered a problem with the bytea type: we can=
't SELECT
>> > bytea strings whose textual representation is too big to fit into St=
ringInfoData.
>> > And as a side effect, pg_dump refuses to dump tables with big bytea =
strings.
>> >=20
>> > It's a bug, it's pretty confusing, but it seems like there's no pret=
ty way
>> > to fix it so far. Here's a link to a recent discussion on the issue:=

>> > https://www.postgresql.org/message-id/flat/c8bdf802d41ec37003ec3b726=
db79428@postgrespro.ru#c8bdf802d41ec37003ec3b726db79428@postgrespro.ru
>> >=20
>> > Since it won't be fixed anytime soon, I thought it could be worth do=
cumenting.
>> > Attaching a patch for the documentation: I added some text to the "B=
inary Data Types"
>> > part where I tried to describe the issue and to explain how to deal =
with it.
>> >=20
>> > My patch in plain text (for convenience):
>> >=20
>> > It is not recommended to use bytea strings whose textual representat=
ion
>> > exceeds 1GB, as it may not be possible to SELECT them due to output =
size
>> > limitations. Consequently, a table containing such big strings canno=
t be
>> > properly processed by pg_dump, as pg_dump will try to SELECT these v=
alues from the
>> > table and fail. The exact size limit advised for bytea strings depen=
ds on their
>> > content, the external format and encoding that you are using, the co=
ntext in
>> > which they will be selected. The general rule is that when you use S=
ELECT,
>> > the returned tuple should not exceed 1GB. Although even if SELECT do=
es not
>> > work, you can still retrieve big bytea strings using COPY in binary =
format.
>>=20
>> Thanks for this message. It took me a while to find out what was the p=
roblem.
>> +1 for documenting this, maybe even with exact error like
>> [ 2020-07-30 01:20:32.248 MSK pg_dump - 10.3.3.30,XX000 ]:ERROR:  inva=
lid memory alloc request size 1472599557
>> It's really really scary. My first feeling was that it's TOAST corrupt=
ion.
>=20
> I still have Anna Akenteva's 2018 email in my mailbox because I wanted
> to research this further.  Now that you have re-reported the problem, I=

> am on it!  ;-)
>=20
> I looked for a clearer reproducible test case, and I have found this
> one:

It is easier to reproduce than that:

select repeat('x',600000000)::bytea;
ERROR:  invalid memory alloc request size 1200000003

select octet_length(repeat('x',600000000)::bytea);
 octet_length
--------------
    600000000
(1 row)

CREATE TABLE big_data AS
 select repeat('x',600000000)::bytea;
SELECT 1

SELECT repeat FROM big_data;
ERROR:  invalid memory alloc request size 1200000003

\q

pg_dump -d postgres > /tmp/a
pg_dump: error: Dumping the contents of table "big_data" failed: PQgetRes=
ult()
failed.
pg_dump: error: Error message from server: ERROR:  invalid memory alloc r=
equest
size 1200000003
pg_dump: error: The command was: COPY public.big_data (repeat) TO stdout;=


There are other cases that also hit the StringInfo limits (i.e. MaxAllocS=
ize)
with TEXT data type (and I guess others, like JSON or XML, too):

select repeat('x',600000000), repeat('y',600000000);
ERROR:  out of memory
DETAIL:  Cannot enlarge string buffer containing 600000010 bytes by 60000=
0000
more bytes.


Joe

--=20
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development


--dpmTzZF8qHq58NHEuWMhn64KWUzWG6nBn--

--EnhV0MYJhJSiTFsOr5HVOtDpWFJSsYG4G
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEg14x9eymXoJyHrH+N/L3QzX32GUFAl8odzYACgkQN/L3QzX3
2GUbVA/+PXWtlaVCCIKGbrdC4C9jdPqktDabgRgZnefkM5dNm4wWCBWI8cqqk2fR
vwRbkpnByDlMAxu8oHP2wyvXf5mgmp8AvfrAWcPavz6Nn/3OltVropM4O0y24fYk
qAxDzXa+fyl0y1LFP3QobiEWH885lQs71imh4v0sS+UcpltY/GxOAf4I4p1Cbf+7
qWUIbCYzXXTXG1bWVwXK8fKxnh7acOGYOp85s6lzrbqWH2CN5AHmaU0mbJlE4iQx
NSAFEymhGOA10MqYNnOZyFhFIWA7MdchniTlZheFIwJRUhhdKFf4Myo/hU+IKopN
erBJkVdsFOpGQIJHfP+4q89FMc5BhpY7uO/YPU1cuHxdLMWYr1DRWPdnlN6vq/d6
TmnZFHc2xPS17N/+XVR9PMuTug/JAl2y9gYL8FK4vjyXjDVzkVCZG9fPgVyg7/lZ
bT4CxxJHmjBFbznw/i0XYsHu5AupPzcntTOhHkpXtz0v2EPAyWJXGMlOH7mdq6NA
TDPdWB5JzJC2VscmvbzxVCgp9Uj3R7K3IifF5cqji9kSssULnTWxk3LjbErsp2ar
Tcl+hTg1HlKoo2uBa7vUSBYxoNnfCFWIVmVrL/HfZxWf4rUJc6CYJKaxvpWqaX5w
eYu2CM1yagK7jV7aHhRBfWanQD3QgZbyBT3eSOCVVueqpsM4vKU=
=UcS2
-----END PGP SIGNATURE-----

--EnhV0MYJhJSiTFsOr5HVOtDpWFJSsYG4G--