public inbox for [email protected]  

help / color / mirror / Atom feed

From: jian he <[email protected]>
To: Joe Conway <[email protected]>
Cc: Andrey M. Borodin <[email protected]>
Cc: Dean Rasheed <[email protected]>
Cc: Daniel Verite <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: Davin Shearer <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Subject: Re: Emitting JSON to file using COPY TO
Date: Fri, 13 Sep 2024 22:42:00 +0800
Message-ID: <CACJufxH=3PreKDHuoskn0GQmxUcw-n0k3ogN1CD4CvFbLBy-VA@mail.gmail.com> (raw)
In-Reply-To: <CACJufxFvOQWcGpiaFkOxMGPc=g85h=-LdyB_2wXBSVQCO9RAHQ@mail.gmail.com>
References: <[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<CAEZATCWh29787xf=4NgkoixeqRHrqi0Qd33Z6_-F8t2dZ0yLCQ@mail.gmail.com>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<CACJufxGMSbpXCCoCb_u5EULo5y2sfMgASkRk2pmw9WRu2qOCXw@mail.gmail.com>
	<CACJufxGjJXKfPZoyFqgvYTBYW3GmB5WA+0H0zXkfYjrc7au7qg@mail.gmail.com>
	<CACJufxET6LJWbywU_odC_k3vYwEKhNQjiWTDk=37ecQG5mFk5Q@mail.gmail.com>
	<CACJufxFvOQWcGpiaFkOxMGPc=g85h=-LdyB_2wXBSVQCO9RAHQ@mail.gmail.com>

Hi.

in ExecutePlan
we have:

    for (;;)
    {
        ResetPerTupleExprContext(estate);
        slot = ExecProcNode(planstate);
        if (!TupIsNull(slot))
        {
            if((slot != NULL) && (slot->tts_tupleDescriptor != NULL)
            && (slot->tts_tupleDescriptor->natts > 0)
            && (slot->tts_tupleDescriptor->attrs->attname.data[0] == '\0'))
            elog(INFO, "%s:%d %s this slot first attribute attname is
null", __FILE_NAME__, __LINE__, __func__);
        }
        if (TupIsNull(slot))
            break;
        if (sendTuples)
        {
            if (!dest->receiveSlot(slot, dest))
                break;
        }


dest->receiveSlot(slot, dest) is responsible for sending values to destination,
for COPY TO it will call copy_dest_receive, CopyOneRowTo.

For the copy to format json, we need to make sure
in "dest->receiveSlot(slot, dest))", the slot->tts_tupleDescriptor has
proper information.
because we *use* slot->tts_tupleDescriptor->attrs->attname as the json key.

For example, if (slot->tts_tupleDescriptor->attrs->attname.data[0] == '\0')
then output json may look like:  {"":12}
which is not what we want.



in ExecutePlan i use
elog(INFO, "%s:%d %s this slot first attribute attname is null",
__FILE_NAME__, __LINE__, __func__);
to find sql queries that attribute name is not good.

based on that, i found out many COPY TO (FORMAT JSON) queries will either
error out or the output json key be empty string
if in CopyOneRowTo we didn't copy the cstate->queryDesc->tupDesc
to the slot->tts_tupleDescriptor


You can test it yourself.
first `git am v12-0001-introduce-json-format-for-COPY-TO.patch`
after that, comment out the memcpy call in CopyOneRowTo, just like the
following:
        if(!cstate->rel)
        {
            // memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0),
            //     TupleDescAttr(cstate->queryDesc->tupDesc, 0),
            //     cstate->queryDesc->tupDesc->natts *
sizeof(FormData_pg_attribute));

build and test with the attached script.
you will see COPY TO FORMAT JSON, lots of cases where the json key
becomes an empty string.


I think this thread related issues has been resolved.


Attachments:

  [application/sql] scratch31.sql (1.5K, 2-scratch31.sql)
  download

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Emitting JSON to file using COPY TO
  In-Reply-To: <CACJufxH=3PreKDHuoskn0GQmxUcw-n0k3ogN1CD4CvFbLBy-VA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox