Message-ID: <c38d850f-bb69-4d7a-b3df-39432ab7f113@dunslane.net>
Date: Sun, 3 Dec 2023 14:47:42 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: Emitting JSON to file using COPY TO
Content-Language: en-US
To: Joe Conway <mail@joeconway.com>, Davin Shearer <davin@apache.org>,
 PostgreSQL-development <pgsql-hackers@postgreSQL.org>
References: <CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU=kcg@mail.gmail.com>
 <fc1e39f8-274f-4be6-8a70-9fb0d9fe84a2@aklaver.com>
 <CAFCRh-_LhP5Ln4-beVz=1pQ9VycU3hAxnJHXhPHZRmX0BSCVEQ@mail.gmail.com>
 <CAFCRh--Vhx0105S7SWAYHGcVMTSRtbcoArDUGSbYSWRUGunW_g@mail.gmail.com>
 <CAKFQuwZb3zaSqkOvGXRRLZEW61az+4xyHZ7tBPR91JfrcdCKvA@mail.gmail.com>
 <3853387.1701096982@sss.pgh.pa.us>
 <CAFCRh-_GdiUvjd5z5FfvTfhruOnYqBu163XU47zZE8RNATCJGQ@mail.gmail.com>
 <3a98decf-3fe3-4b49-9b68-fda01338872c@sedlakovi.org>
 <CALvfUkCq3pvkmniWEKZwkbA6hNNvkANF3OamMfhC170RgY+HJg@mail.gmail.com>
 <24e3ee88-ec1e-421b-89ae-8a47ee0d2df1@joeconway.com>
 <b22367ef-6ac1-e1c0-da05-0e760e1b7e29@dunslane.net>
 <8c88da85-c197-4765-96f8-a9a1c78305a6@joeconway.com>
 <f875d187-b9a8-4def-896c-a399d05a8723@joeconway.com>
From: Andrew Dunstan <andrew@dunslane.net>
In-Reply-To: <f875d187-b9a8-4def-896c-a399d05a8723@joeconway.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/c38d850f-bb69-4d7a-b3df-39432ab7f113%40dunslane.net>
Precedence: bulk


On 2023-12-03 Su 12:11, Joe Conway wrote:
> On 12/3/23 11:03, Joe Conway wrote:
>>   From your earlier post, regarding constructing the aggregate -- not
>> extensive testing but one data point:
>> 8<--------------------------
>> test=# copy foo to '/tmp/buf' (format json, force_array);
>> COPY 10000000
>> Time: 36353.153 ms (00:36.353)
>> test=# copy (select json_agg(foo) from foo) to '/tmp/buf';
>> COPY 1
>> Time: 46835.238 ms (00:46.835)
>> 8<--------------------------
>
> Also if the table is large enough, the aggregate method is not even 
> feasible whereas the COPY TO method works:
> 8<--------------------------
> test=# select count(*) from foo;
>   count
> ----------
>  20000000
> (1 row)
>
> test=# copy (select json_agg(foo) from foo) to '/tmp/buf';
> ERROR:  out of memory
> DETAIL:  Cannot enlarge string buffer containing 1073741822 bytes by 1 
> more bytes.
>
> test=# copy foo to '/tmp/buf' (format json, force_array);
> COPY 20000000
> 8<--------------------------


None of this is surprising. As I mentioned, limitations with json_agg() 
are why I support the idea of this patch.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com