From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: John Naylor <john.naylor@enterprisedb.com>
To: jitesh tiwari <jitesh120@gmail.com>
Cc: pgsql-bugs@postgresql.org
Subject: Re: fetching bytea (blob) data of 850 MB from psql client failed
Date: Wed, 31 Aug 2022 15:20:29 +0200
Message-ID: <e40f0c2d-68c7-dafd-2628-7eca2c62b425@enterprisedb.com> (raw)
In-Reply-To: <CAFBsxsG=UcZo_pM2kLho3qPv67YBaWa7GuQ6iWb1g-5gLf+8BA@mail.gmail.com>
References: <CAOEdJKF4PFrHtH0OEn+Qmeqy-Z7UnNWUMAtrT+jVhTo3ozUjxw@mail.gmail.com>
<CAFBsxsExRqZjvkMNBLgS8D-5BUBG6JYhMsqngy=DB5cZOYBV=A@mail.gmail.com>
<CAOEdJKGWw1_f5HvN6YD=0XECRRjAw1Hp5hb=AmTxoVe04VTHKA@mail.gmail.com>
<CAFBsxsG=UcZo_pM2kLho3qPv67YBaWa7GuQ6iWb1g-5gLf+8BA@mail.gmail.com>
On 8/31/22 09:27, John Naylor wrote:
> On Mon, Aug 29, 2022 at 1:02 PM jitesh tiwari <jitesh120@gmail.com> wrote:
>>
>> Hi John,
>> Thanks for those details.
>> The database table for bytea columns has binary data data and some are not text. Hence, when it tried to fetch by changing the bytea_output to 'escape' it failed again with the below error -
>> VERROR; result of bytea output conversion is too large(File varlena.c; Line 415; Routine byteaout; ).
>> BTW, is there any plan to fix this issue in the PostgreSQL backend code in near future?
>
> Unfortunately, there is no plan as there is no consensus on how to
> approach the issue. It seems the possible actions are (easiest to
> hardest):
>
> 1. Document that bytea is unsafe -- you can insert anything you like,
> but you may be unable to retrieve it again.
> 2. Determine some maximum size that guarantees accessibility and
> invent a GUC that by default prevents inserting larger values than
> that.
> 3. Fix the issue properly.
>
> #2 and #3 are challenging for reasons given in the thread I linked to above.
>
I haven't tried, but wouldn't it be enough to fetch the data in smaller
chunks? The application would have to re-assemble that, of course.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tomas.vondra@enterprisedb.com, john.naylor@enterprisedb.com, jitesh120@gmail.com
Subject: Re: fetching bytea (blob) data of 850 MB from psql client failed
In-Reply-To: <e40f0c2d-68c7-dafd-2628-7eca2c62b425@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox