Message-ID: <e40f0c2d-68c7-dafd-2628-7eca2c62b425@enterprisedb.com>
Date: Wed, 31 Aug 2022 15:20:29 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: fetching bytea (blob) data of 850 MB from psql client failed
Content-Language: en-US
To: John Naylor <john.naylor@enterprisedb.com>,
 jitesh tiwari <jitesh120@gmail.com>
Cc: pgsql-bugs@postgresql.org
References: 
 <CAOEdJKF4PFrHtH0OEn+Qmeqy-Z7UnNWUMAtrT+jVhTo3ozUjxw@mail.gmail.com>
 <CAFBsxsExRqZjvkMNBLgS8D-5BUBG6JYhMsqngy=DB5cZOYBV=A@mail.gmail.com>
 <CAOEdJKGWw1_f5HvN6YD=0XECRRjAw1Hp5hb=AmTxoVe04VTHKA@mail.gmail.com>
 <CAFBsxsG=UcZo_pM2kLho3qPv67YBaWa7GuQ6iWb1g-5gLf+8BA@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: 
 <CAFBsxsG=UcZo_pM2kLho3qPv67YBaWa7GuQ6iWb1g-5gLf+8BA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/e40f0c2d-68c7-dafd-2628-7eca2c62b425%40enterprisedb.com>
Precedence: bulk


On 8/31/22 09:27, John Naylor wrote:
> On Mon, Aug 29, 2022 at 1:02 PM jitesh tiwari <jitesh120@gmail.com> wrote:
>>
>> Hi John,
>> Thanks for those details.
>> The database table for bytea columns has binary data data and some are not text. Hence, when it tried to fetch by changing the bytea_output to 'escape' it failed again with the below error -
>> VERROR; result of bytea output conversion is too large(File varlena.c; Line 415; Routine byteaout; ).
>> BTW, is there any plan to fix this issue in the PostgreSQL backend code in near future?
> 
> Unfortunately, there is no plan as there is no consensus on how to
> approach the issue. It seems the possible actions are (easiest to
> hardest):
> 
> 1. Document that bytea is unsafe  -- you can insert anything you like,
> but you may be unable to retrieve it again.
> 2. Determine some maximum size that guarantees accessibility and
> invent a GUC that by default prevents inserting larger values than
> that.
> 3. Fix the issue properly.
> 
> #2 and #3 are challenging for reasons given in the thread I linked to above.
> 

I haven't tried, but wouldn't it be enough to fetch the data in smaller
chunks? The application would have to re-assemble that, of course.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company