Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rB4MA-009Juh-Iv for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Dec 2023 02:43:19 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rB4M5-00EOeQ-3y for pgsql-hackers@arkaria.postgresql.org; Thu, 07 Dec 2023 02:43:13 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rB4M3-00EOeD-Fh for pgsql-hackers@lists.postgresql.org; Thu, 07 Dec 2023 02:43:12 +0000 Received: from out1-smtp.messagingengine.com ([66.111.4.25]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rB4Ly-00AUBg-Sc for pgsql-hackers@postgresql.org; Thu, 07 Dec 2023 02:43:10 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 6E3BD5C0162; Wed, 6 Dec 2023 21:43:01 -0500 (EST) Received: from imap49 ([10.202.2.99]) by compute2.internal (MEProxy); Wed, 06 Dec 2023 21:43:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eulerto.com; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1701916981; x=1702003381; bh=f+ DDvNxhL8GTXAABIai8G/eaNXcVYhM1LDPCg9Aps1Y=; b=Zw5HvF1eUfv+wCVwuT Uy051j1KrZ7n1i0155QiGXdhF1GaKJ5YsXB/9hhpfNmxnIEgJmakfof3oRO0ieXq 3q+VGdm0qzI9S7j3mJolRI91flg36NL3k+L3hBURJGBE/s5opRRYJsqbKixQcJIp kZ8c8OSq+drlKSJQ7UaZHgROype+l2gr13QgrE8a7IdJ7XqyZmo15JfpPgZrtOtQ mJ90HY3h5+Xq6VC9EYM+I5OHRH2sg+TC6Q7vcj+AOPJqmj++pbXTH82tHruwVF5H Dpl3r8R72+pcVD1toIoi6SToYxO9OoyQRgT4Jqt4isunwM0cwA6/o2aigGeJjJB6 oMKg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1701916981; x=1702003381; bh=f+DDvNxhL8GTX AABIai8G/eaNXcVYhM1LDPCg9Aps1Y=; b=fDt4sVRFf8ZNcidB4ThCR8KyPNypD Ofhjx6G/me7BGno+EFDWzH0Ft89eMDOXR374VTmRf968PTsEUS/Oh1lN1K0+E9HK tENVyXw4yWYkFr/HcXTrNZ2Bcwz0CtfLL5ZoyjNbrTwORwC5uNqKsINOhuEvL0qu rk0bFn0inl4uixDknvxlp7vzXoLcxDY0cqu2WG+gcQwcZuOuuW2aFsbpb3wnNeIg 7xnSfg2n2dki4O7ulb5OQe+4GLzKIhrd8EzSu03UykTyQ/dKaOmYttFRn/ZV1usg 0OHp5BrT4WcoYC8bZRghlnrEKGoFC9KjwwEFl6dSVh+kZ+PQBcJYtXAnw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrudekuddggeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerredtnecuhfhrohhmpedfgfhu lhgvrhcuvfgrvhgvihhrrgdfuceovghulhgvrhesvghulhgvrhhtohdrtghomheqnecugg ftrfgrthhtvghrnhepgfffjedviedvkedviedvteduieekudeivedthffhuddvteegleet ieffkeeghfdvnecuffhomhgrihhnpegvnhhtvghrphhrihhsvggusgdrtghomhenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegvuhhlvghrsegv uhhlvghrthhordgtohhm X-ME-Proxy: Feedback-ID: i0c21471d:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 99F7515A0091; Wed, 6 Dec 2023 21:43:00 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1178-geeaf0069a7-fm-20231114.001-geeaf0069 MIME-Version: 1.0 Message-Id: <6ab82c2f-c19f-4cd6-a2a9-ddf9832e6209@app.fastmail.com> In-Reply-To: References: Date: Wed, 06 Dec 2023 23:42:06 -0300 From: "Euler Taveira" To: "Daniel Verite" , "Andrew Dunstan" Cc: "Joe Conway" , "Davin Shearer" , pgsql-hackers Subject: Re: Emitting JSON to file using COPY TO Content-Type: multipart/alternative; boundary=494471d1822540a190ecee42e1dde96b List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --494471d1822540a190ecee42e1dde96b Content-Type: text/plain On Wed, Dec 6, 2023, at 3:59 PM, Daniel Verite wrote: > The first Copy data message with contents "5b0a" does not qualify > as a row of data with 3 columns as advertised in the CopyOut > message. Isn't that a problem? > > At least the json non-ARRAY case ("json lines") doesn't have > this issue, since every CopyData message corresponds effectively > to a row in the table. Moreover, if your interface wants to process the COPY data stream while receiving it, you cannot provide "json array" format because each row (plus all of the received ones) is not a valid JSON. Hence, a JSON parser cannot be executed until you receive the whole data set. (wal2json format 1 has this disadvantage. Format 2 was born to provide a better alternative -- each row is a valid JSON.) I'm not saying that "json array" is not useful but that for large data sets, it is less useful. -- Euler Taveira EDB https://www.enterprisedb.com/ --494471d1822540a190ecee42e1dde96b Content-Type: text/html Content-Transfer-Encoding: quoted-printable
On Wed, Dec 6, = 2023, at 3:59 PM, Daniel Verite wrote:
The first Copy data message with contents "5= b0a" does not qualify
as a row of data with 3 columns as a= dvertised in the CopyOut
message. Isn't that a problem?

At least the json non-ARRAY case ("json lines= ") doesn't have
this issue, since every CopyData message c= orresponds effectively
to a row in the table.

Moreover, if your interface wants to proce= ss the COPY data stream while
receiving it, you cannot pro= vide "json array" format because each row (plus all
of the= received ones) is not a valid JSON. Hence, a JSON parser cannot be
<= /div>
executed until you receive the whole data set. (wal2json forma= t 1 has this
disadvantage. Format 2 was born to provide a = better alternative -- each row is
a valid JSON.) I'm not s= aying that "json array" is not useful but that for
large d= ata sets, it is less useful.


--
Euler Taveira
EDB &nbs= p; https://www.enterprisedb.co= m/

--494471d1822540a190ecee42e1dde96b--