MIME-Version: 1.0
References: <b3a73eca-ec3b-4b64-b6c2-82418bb16701@manitou-mail.org>
 <CACJufxGq6aJszBhfrUQKHeKmp0r+ka7=SwHSY7Qs_6LQMwiEbw@mail.gmail.com>
 <CA+v5N43-F9-Wiktg_-+aqKJz+YiCNJMAh5ootDeTKkOJ=kiaHA@mail.gmail.com>
 <CACJufxEjZwrocCpt29xtmJTwhWZUu1Nt0GfHFarNGXUS5AAecw@mail.gmail.com>
 <CAEG8a3LB7q1eQ0AzFTEBDFDxs3kJ=5iJA+HTmiYGya6Wb5jsRA@mail.gmail.com>
 <CACJufxFLpMsbW41T65xJsw925MSE1bvOr6X+h_7sw8_qmDpTpA@mail.gmail.com>
 <74d3b4e2-d201-489d-9b9a-1a6d0eb492a6@dunslane.net> <6cbbf7f3-da4a-4201-98a6-4347a3c35db0@joeconway.com>
 <8154f8e2-7d9c-442f-a7cb-a5caf38f6a12@dunslane.net> <CACJufxELGPA86veVNNBoqUDLwwQbQKdiYLKtwLb7DoSgV6f8sw@mail.gmail.com>
 <57db4b6e-51ba-4984-9343-bc62a3ee0b7c@dunslane.net> <CACJufxFFZqxC3p4WjpTEi4riaJm=pADX+py0yQ0=RWTn5cqK3Q@mail.gmail.com>
 <f137f5d7-bae7-4145-8c76-e83d2de13d59@dunslane.net> <CACJufxGm3OhbG3=bj4nLxMzq92A84v6Q1dG+hR_ZFrfLg1B99w@mail.gmail.com>
 <CAD21AoASG2uXJXN=23jkSgKWJNVz2TPazQmrOYA7ZvO3GxjVhw@mail.gmail.com> <964a3377-d957-4a06-ad02-8228aefa65fc@dunslane.net>
In-Reply-To: <964a3377-d957-4a06-ad02-8228aefa65fc@dunslane.net>
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Tue, 17 Mar 2026 10:50:35 -0700
Message-ID: <CAD21AoCFGwU3pvvFeFW-NYnJX_QKZ2xKgNEH__ho_a-5aN__iA@mail.gmail.com>
Subject: Re: Emitting JSON to file using COPY TO
To: Andrew Dunstan <andrew@dunslane.net>
Cc: jian he <jian.universality@gmail.com>, Joe Conway <mail@joeconway.com>, 
	Junwang Zhao <zhjwpku@gmail.com>, Florents Tselai <florents.tselai@gmail.com>, 
	"Andrey M. Borodin" <x4mmm@yandex-team.ru>, Dean Rasheed <dean.a.rasheed@gmail.com>, 
	Daniel Verite <daniel@manitou-mail.org>, Davin Shearer <davin@apache.org>, 
	PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAD21AoCFGwU3pvvFeFW-NYnJX_QKZ2xKgNEH__ho_a-5aN__iA%40mail.gmail.com>
Precedence: bulk

On Mon, Mar 16, 2026 at 1:59=E2=80=AFPM Andrew Dunstan <andrew@dunslane.net=
> wrote:
>
>
> On 2026-03-16 Mo 2:24 PM, Masahiko Sawada wrote:
>
> On Sun, Mar 8, 2026 at 8:49=E2=80=AFPM jian he <jian.universality@gmail.c=
om> wrote:
>
> On Mon, Mar 9, 2026 at 3:44=E2=80=AFAM Andrew Dunstan <andrew@dunslane.ne=
t> wrote:
>
> Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that?
> How about something like this?:
>
> -        * Full table or query without column list.  Ensure the slot uses
> -        * cstate->tupDesc so that the datum is stamped with the right ty=
pe;
> -        * for queries output type is RECORDOID this must be the blessed
> -        * descriptor so that composite_to_json can look it up via
> -        * lookup_rowtype_tupdesc.
> +        * Full table or query without column list.  For queries, the slo=
t's
> +        * TupleDesc may carry RECORDOID, which is not registered in the
> type
> +        * cache and would cause composite_to_json's lookup_rowtype_tupde=
sc
> +        * call to fail.  Build a HeapTuple stamped with the blessed
> +        * descriptor so the type can be looked up correctly.
>           */
>          if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid =3D=3D
> RECORDOID)
> -           slot->tts_tupleDescriptor =3D cstate->queryDesc->tupDesc;
> +       {
> +           HeapTuple   tup;
>
> -       rowdata =3D ExecFetchSlotHeapTupleDatum(slot);
> +           tup =3D heap_form_tuple(cstate->tupDesc,
> +                                 slot->tts_values,
> +                                 slot->tts_isnull);
> +           rowdata =3D HeapTupleGetDatum(tup);
> +       }
> +       else
> +       {
> +           rowdata =3D ExecFetchSlotHeapTupleDatum(slot);
> +       }
>
> This is better. I've tried to get rid of json_projvalues and json_projnul=
ls.
> Just using heap_form_tuple, but it won't work.
>
> I incorporated the v28-0004 COPY column list into v9-0002.
> With this patch set, we added four fields to the struct CopyToStateData.
>
> +    StringInfo    json_buf;        /* reusable buffer for JSON output,
> +                                 * initialized in BeginCopyTo */
> +    TupleDesc    tupDesc;        /* Descriptor for JSON output; for a co=
lumn
> +                                 * list this is a projected descriptor *=
/
> +    Datum       *json_projvalues;    /* pre-allocated projection values,=
 or
> +                                     * NULL */
> +    bool       *json_projnulls; /* pre-allocated projection nulls, or NU=
LL */
>
> Using the script in
> https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX=
%2Bpy0yQ0%3DRWTn5cqK3Q%40mail.gmail.com
> I tested it again on macOS and Linux, and there are no regressions for
> COPY TO with the TEXT and CSV formats.
>
> I've reviewed the patch and have some comments:
>
> ---
> I got a SEGV in the following scenario:
>
> postgres(1:1197708)=3D# create table test (a int, b text, c jsonb);
> CREATE TABLE
> postgres(1:1197708)=3D# copy test(a, b) to stdout with (format 'json' );
> TRAP: failed Assert("tupdesc->firstNonCachedOffsetAttr >=3D 0"), File:
> "execTuples.c", Line: 2328, PID: 1197708
> postgres: masahiko postgres [local] COPY(ExceptionalCondition+0x9e) [0xbe=
be48]
> postgres: masahiko postgres [local] COPY(BlessTupleDesc+0x2b) [0x729b50]
> postgres: masahiko postgres [local] COPY(BeginCopyTo+0xc94) [0x637bdf]
> postgres: masahiko postgres [local] COPY(DoCopy+0xb68) [0x62afbc]
> postgres: masahiko postgres [local]
> COPY(standard_ProcessUtility+0xa22) [0xa0ba48]
> postgres: masahiko postgres [local] COPY(ProcessUtility+0x10e) [0xa0b01f]
> postgres: masahiko postgres [local] COPY() [0xa09872]
> postgres: masahiko postgres [local] COPY() [0xa09acf]
> postgres: masahiko postgres [local] COPY(PortalRun+0x2c8) [0xa0901d]
> postgres: masahiko postgres [local] COPY() [0xa02055]
> postgres: masahiko postgres [local] COPY(PostgresMain+0xaf1) [0xa0724e]
> postgres: masahiko postgres [local] COPY() [0x9fdab9]
> postgres: masahiko postgres [local]
> COPY(postmaster_child_launch+0x165) [0x905378]
> postgres: masahiko postgres [local] COPY() [0x90b600]
> postgres: masahiko postgres [local] COPY() [0x908e6a]
> postgres: masahiko postgres [local] COPY(PostmasterMain+0x14fe) [0x90880c=
]
> postgres: masahiko postgres [local] COPY(main+0x340) [0x7a1f9c]
>
> It seems to forget to call TupleDescFinalize(). And I think we need
> some regression tests for this case.
>
> ---
> +       if (cstate->opts.format =3D=3D COPY_FORMAT_JSON)
> +       {
> +               /*
> +                * If FORCE_ARRAY has been specified, send the opening br=
acket.
> +                */
> +               if (cstate->opts.force_array)
> +               {
> +                       CopySendChar(cstate, '[');
> +                       CopySendTextLikeEndOfRow(cstate);
> +               }
> +       }
>
> We can conjunct the two if statement conditions.
>
>
> Here's a v30 set that I hope fixes these issues.

Thank you for updating the patch! The patches look good to me.

Regards,

--=20
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com