public inbox for [email protected]help / color / mirror / Atom feed
Re: Emitting JSON to file using COPY TO 22+ messages / 7 participants [nested] [flat]
* Re: Emitting JSON to file using COPY TO @ 2025-11-10 00:53 jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2025-11-10 00:53 UTC (permalink / raw) To: Junwang Zhao <[email protected]>; +Cc: Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Wed, Oct 1, 2025 at 2:16 PM jian he <[email protected]> wrote: > > hi. > v19 attached, same as v18. > repost it so that CFbot can pick up the latest patchset. hi. new patch attached, rebase only. Attachments: [text/x-patch] v20-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 2-v20-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From 97e63d2b7de1fef820305b279d9e5602c82dab53 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 10 Nov 2025 08:39:36 +0800 Subject: [PATCH v20 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans, binary and csv_mode, within CopyFormatOptions. This approach, while functional, isn't ideal for future expansion. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 28e878c3688..d674ada98e4 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -564,6 +564,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = (CopyFormatOptions *) palloc0(sizeof(CopyFormatOptions)); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -578,11 +580,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -742,31 +744,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -814,7 +816,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -822,43 +824,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -872,8 +874,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -888,8 +890,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -913,7 +915,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -949,7 +951,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -966,7 +968,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 12781963b4f..ba31b227d5f 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -155,9 +155,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -261,7 +261,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index b1ae97b833d..578e6c0c9a2 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -171,7 +171,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -747,7 +747,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -774,7 +774,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index cef452584e5..c97f0460b3e 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 541176e1980..686653233b2 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -48,6 +48,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -58,9 +68,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 432509277c9..256b5000af4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -516,6 +516,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 [text/x-patch] v20-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (9.7K, 3-v20-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From e42e36118f73fa1a98f698031e3f1f7cbb9150cf Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 30 Jul 2025 19:50:41 +0800 Subject: [PATCH v20 3/3] Add option force_array for COPY JSON FORMAT force_array option can only be used in COPY TO with JSON format. it make the output json output behave like json array type. refactored by Junwang Zhao to adapt the newly introduced CopyToRoutine struct(2e4127b6d2). Author: Joe Conway <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 14 +++++++++++ src/backend/commands/copy.c | 13 +++++++++++ src/backend/commands/copyto.c | 37 +++++++++++++++++++++++++++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 +++++++++++++++++++ src/test/regress/sql/copy.sql | 8 +++++++ 7 files changed, 96 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 2d6d6802cbd..d8d9fb173b4 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>FORCE_QUOTE</literal></term> <listitem> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 0ec9b22d20f..6f9ae3fbfd7 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -557,6 +557,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -713,6 +714,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -968,6 +976,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index accf34e1a60..b58c5bdf987 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,10 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* need delimiter to start next json array element */ + bool json_row_delim_needed; + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -133,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +182,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +248,16 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If JSON has been requested, and FORCE_ARRAY has been specified send the + * opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -354,11 +369,31 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) result = makeStringInfo(); composite_to_json(rowdata, result, false); + if (cstate->json_row_delim_needed && cstate->opts.force_array) + CopySendChar(cstate, ','); + else if (cstate->opts.force_array) + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + CopySendData(cstate, result->data, result->len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 0fd06a31201..e550aa38a25 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 85aedc267d6..7274b0d3ca5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,6 +87,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 10333357d68..8becc70ee7a 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -112,6 +112,29 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); ERROR: COPY json mode cannot be used with COPY FROM -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only used with JSON mode +--ok +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 80799e2ead9..6a14cfc6e68 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -101,6 +101,14 @@ copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); + +--ok +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v20-0002-json-format-for-COPY-TO.patch (22.2K, 4-v20-0002-json-format-for-COPY-TO.patch) download | inline diff: From 089630f0f4706b31c4c886d033e15ad20761dd6d Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 10 Nov 2025 08:40:26 +0800 Subject: [PATCH v20 2/3] json format for COPY TO JSON format is only supported with the COPY TO operation. It is incompatible with options such as HEADER, DEFAULT, NULL, DELIMITER, and several others. This has been thoroughly tested in src/test/regress/sql/copy.sql The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues, and further adapted by Junwang Zhao to support the newly introduced CopyToRoutine struct (commit 2e4127b6d2). Author: Joe Conway <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 72 +++++++++++++++++++++------- src/backend/commands/copyto.c | 76 ++++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 ++++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 76 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 47 ++++++++++++++++++ 10 files changed, 268 insertions(+), 34 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index fdc24b36bb8..2d6d6802cbd 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index d674ada98e4..0ec9b22d20f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -585,6 +585,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -744,21 +746,42 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + if (opts_out->delim) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DELIMITER")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + if (opts_out->null_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "NULL")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "NULL")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + if (opts_out->default_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "DEFAULT")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DEFAULT")); + } /* Set defaults for omitted options */ if (!opts_out->delim) @@ -824,11 +847,18 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + if (opts_out->header_line != COPY_HEADER_FALSE) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "HEADER")); + else if(opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot specify %s in JSON mode", "HEADER")); + } /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -932,6 +962,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index c97f0460b3e..accf34e1a60 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -130,6 +132,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +152,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV and text, json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +319,46 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + StringInfo result; + + /* + * If COPY TO source data come from query rather than plain table, we need + * copy CopyToState->QueryDesc->TupleDesc to slot->tts_tupleDescriptor. + * This is necessary because the slot's TupleDesc may change during query + * execution, and we depend on it when calling composite_to_json. + */ + if (!cstate->rel) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + } + rowdata = ExecFetchSlotHeapTupleDatum(slot); + result = makeStringInfo(); + composite_to_json(rowdata, result, false); + + CopySendData(cstate, result->data, result->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +450,21 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * JSON format is always one non-binary column + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -504,7 +564,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 57fe0186547..7cbdadc98ac 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3557,6 +3557,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3639,6 +3643,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 06dd62f0008..647adafd227 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 316a2dafbf1..0fd06a31201 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3376,7 +3376,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO [PROGRAM] <sth> WITH (FORMAT */ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAnyExcept("PROGRAM"), "WITH", "(", "FORMAT") || Matches("COPY|\\copy", MatchAny, "FROM|TO", "PROGRAM", MatchAny, "WITH", "(", "FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM [PROGRAM] filename WITH (ON_ERROR */ else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAnyExcept("PROGRAM"), "WITH", "(", "ON_ERROR") || diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 686653233b2..85aedc267d6 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -56,6 +56,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index 49bbda7ac06..1fa8e2ce8e2 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 24e0f472f14..10333357d68 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,82 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY json mode cannot be used with COPY FROM +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 676a8b342b5..80799e2ead9 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,53 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2025-11-29 02:46 jian he <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2025-11-29 02:46 UTC (permalink / raw) To: Junwang Zhao <[email protected]>; +Cc: Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> hi. https://commitfest.postgresql.org/patch/4716/ says need rebase. turns out only copy.sgml has conflict. it conflict with https://git.postgresql.org/cgit/postgresql.git/commit/?id=e4018f891dec09ff95ac97e5b1a2307349aeeffa So the rebase only needs to update copy.sgml. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v21-0002-json-format-for-COPY-TO.patch (22.2K, 2-v21-0002-json-format-for-COPY-TO.patch) download | inline diff: From 3ff9058b29e5410a976c1013484f55df27f132a5 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 10 Nov 2025 08:40:26 +0800 Subject: [PATCH v21 2/3] json format for COPY TO JSON format is only supported with the COPY TO operation. It is incompatible with options such as HEADER, DEFAULT, NULL, DELIMITER, and several others. This has been thoroughly tested in src/test/regress/sql/copy.sql The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues, and further adapted by Junwang Zhao to support the newly introduced CopyToRoutine struct (commit 2e4127b6d2). Author: Joe Conway <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 72 +++++++++++++++++++++------- src/backend/commands/copyto.c | 76 ++++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 ++++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 76 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 47 ++++++++++++++++++ 10 files changed, 268 insertions(+), 34 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 53b0ea8f573..320f5f1edd4 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index d674ada98e4..0ec9b22d20f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -585,6 +585,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -744,21 +746,42 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + if (opts_out->delim) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DELIMITER")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + if (opts_out->null_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "NULL")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "NULL")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + if (opts_out->default_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "DEFAULT")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DEFAULT")); + } /* Set defaults for omitted options */ if (!opts_out->delim) @@ -824,11 +847,18 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + if (opts_out->header_line != COPY_HEADER_FALSE) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "HEADER")); + else if(opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot specify %s in JSON mode", "HEADER")); + } /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -932,6 +962,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index c97f0460b3e..accf34e1a60 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -130,6 +132,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +152,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV and text, json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +319,46 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + StringInfo result; + + /* + * If COPY TO source data come from query rather than plain table, we need + * copy CopyToState->QueryDesc->TupleDesc to slot->tts_tupleDescriptor. + * This is necessary because the slot's TupleDesc may change during query + * execution, and we depend on it when calling composite_to_json. + */ + if (!cstate->rel) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + } + rowdata = ExecFetchSlotHeapTupleDatum(slot); + result = makeStringInfo(); + composite_to_json(rowdata, result, false); + + CopySendData(cstate, result->data, result->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +450,21 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * JSON format is always one non-binary column + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -504,7 +564,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index c3a0a354a9c..dd23c075710 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3557,6 +3557,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3639,6 +3643,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 06dd62f0008..647adafd227 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 20d7a65c614..2029b1930c8 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3377,7 +3377,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO [PROGRAM] <sth> WITH (FORMAT */ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAnyExcept("PROGRAM"), "WITH", "(", "FORMAT") || Matches("COPY|\\copy", MatchAny, "FROM|TO", "PROGRAM", MatchAny, "WITH", "(", "FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM [PROGRAM] filename WITH (ON_ERROR */ else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAnyExcept("PROGRAM"), "WITH", "(", "ON_ERROR") || diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 686653233b2..85aedc267d6 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -56,6 +56,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index 49bbda7ac06..1fa8e2ce8e2 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 24e0f472f14..10333357d68 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,82 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY json mode cannot be used with COPY FROM +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 676a8b342b5..80799e2ead9 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,53 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 [text/x-patch] v21-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (9.7K, 3-v21-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 93b28c6521d5ea6f7a4e901acf910f19e8ef6528 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Sat, 29 Nov 2025 10:37:01 +0800 Subject: [PATCH v21 3/3] Add option force_array for COPY JSON FORMAT force_array option can only be used in COPY TO with JSON format. it make the output json output behave like json array type. refactored by Junwang Zhao to adapt the newly introduced CopyToRoutine struct(2e4127b6d2). Author: Joe Conway <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 14 +++++++++++ src/backend/commands/copy.c | 13 +++++++++++ src/backend/commands/copyto.c | 37 +++++++++++++++++++++++++++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 +++++++++++++++++++ src/test/regress/sql/copy.sql | 8 +++++++ 7 files changed, 96 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 320f5f1edd4..a274118b0fa 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 0ec9b22d20f..6f9ae3fbfd7 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -557,6 +557,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -713,6 +714,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -968,6 +976,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index accf34e1a60..b58c5bdf987 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,10 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* need delimiter to start next json array element */ + bool json_row_delim_needed; + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -133,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +182,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +248,16 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If JSON has been requested, and FORCE_ARRAY has been specified send the + * opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -354,11 +369,31 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) result = makeStringInfo(); composite_to_json(rowdata, result, false); + if (cstate->json_row_delim_needed && cstate->opts.force_array) + CopySendChar(cstate, ','); + else if (cstate->opts.force_array) + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + CopySendData(cstate, result->data, result->len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 2029b1930c8..c25c3cdb72b 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 85aedc267d6..7274b0d3ca5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,6 +87,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 10333357d68..8becc70ee7a 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -112,6 +112,29 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); ERROR: COPY json mode cannot be used with COPY FROM -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only used with JSON mode +--ok +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 80799e2ead9..6a14cfc6e68 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -101,6 +101,14 @@ copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); + +--ok +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v21-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 4-v21-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From fbd5ef17532bc111e40d9352049fe9fd4e807aae Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 10 Nov 2025 08:39:36 +0800 Subject: [PATCH v21 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans, binary and csv_mode, within CopyFormatOptions. This approach, while functional, isn't ideal for future expansion. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 28e878c3688..d674ada98e4 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -564,6 +564,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = (CopyFormatOptions *) palloc0(sizeof(CopyFormatOptions)); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -578,11 +580,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -742,31 +744,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -814,7 +816,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -822,43 +824,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -872,8 +874,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -888,8 +890,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -913,7 +915,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -949,7 +951,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -966,7 +968,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 12781963b4f..ba31b227d5f 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -155,9 +155,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -261,7 +261,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index a09e7fbace3..9cc5bb1cf5c 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -171,7 +171,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -747,7 +747,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -774,7 +774,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index cef452584e5..c97f0460b3e 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 541176e1980..686653233b2 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -48,6 +48,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -58,9 +68,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index cf3f6a7dafd..6d1f2d5bb3b 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -516,6 +516,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-01-21 12:15 jian he <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2026-01-21 12:15 UTC (permalink / raw) To: Junwang Zhao <[email protected]>; +Cc: Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> hi. rebase only. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v22-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 2-v22-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From c47611ebcafd3e8f58f1b1698aff8289c99b5899 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v22 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 9c51384ab92..8d92c513fad 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -564,6 +564,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -578,11 +580,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -742,31 +744,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -814,7 +816,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -822,43 +824,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -872,8 +874,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -888,8 +890,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -913,7 +915,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -949,7 +951,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -966,7 +968,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 25ee20b23db..0c73b0259c3 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -155,9 +155,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -261,7 +261,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 5868a7fa11f..a28e7ee3626 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -171,7 +171,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -747,7 +747,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -774,7 +774,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 4ab4a3893d5..880231ea6cd 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 43c2580539f..7be41d9da12 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -48,6 +48,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -58,9 +68,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3f3a888fd0e..03fae82f656 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -527,6 +527,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 [text/x-patch] v22-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (9.7K, 3-v22-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 366db5fc956ea2436cc16f4dc7946411ffe0f2b3 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Sat, 29 Nov 2025 10:37:01 +0800 Subject: [PATCH v22 3/3] Add option force_array for COPY JSON FORMAT force_array option can only be used in COPY TO with JSON format. it make the output json output behave like json array type. refactored by Junwang Zhao to adapt the newly introduced CopyToRoutine struct(2e4127b6d2). Author: Joe Conway <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 14 +++++++++++ src/backend/commands/copy.c | 13 +++++++++++ src/backend/commands/copyto.c | 37 +++++++++++++++++++++++++++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 +++++++++++++++++++ src/test/regress/sql/copy.sql | 8 +++++++ 7 files changed, 96 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 320f5f1edd4..a274118b0fa 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7860afe44da..c2b432a96eb 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -557,6 +557,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -713,6 +714,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -968,6 +976,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index f3f257eaef2..a72bfed9814 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,10 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* need delimiter to start next json array element */ + bool json_row_delim_needed; + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -133,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +182,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +248,16 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If JSON has been requested, and FORCE_ARRAY has been specified send the + * opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -354,11 +369,31 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) result = makeStringInfo(); composite_to_json(rowdata, result, false); + if (cstate->json_row_delim_needed && cstate->opts.force_array) + CopySendChar(cstate, ','); + else if (cstate->opts.force_array) + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + CopySendData(cstate, result->data, result->len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 37cb64d2b72..a610ef603f4 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index fe85f02b110..b2a68fa6407 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,6 +87,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 10333357d68..8becc70ee7a 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -112,6 +112,29 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); ERROR: COPY json mode cannot be used with COPY FROM -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only used with JSON mode +--ok +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 80799e2ead9..6a14cfc6e68 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -101,6 +101,14 @@ copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); + +--ok +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v22-0002-json-format-for-COPY-TO.patch (21.9K, 4-v22-0002-json-format-for-COPY-TO.patch) download | inline diff: From 861cf905fcddece82a13aaff4b400992deb881f2 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:43:14 +0800 Subject: [PATCH v22 2/3] json format for COPY TO JSON format is only supported with the COPY TO operation. It is incompatible with options such as HEADER, DEFAULT, NULL, DELIMITER, and several others. This has been thoroughly tested in src/test/regress/sql/copy.sql The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues, and further adapted by Junwang Zhao to support the newly introduced CopyToRoutine struct (commit 2e4127b6d2). Author: Joe Conway <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 72 +++++++++++++++++++++------- src/backend/commands/copyto.c | 76 ++++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 ++++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 76 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 47 ++++++++++++++++++ 10 files changed, 268 insertions(+), 34 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 53b0ea8f573..320f5f1edd4 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 8d92c513fad..7860afe44da 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -585,6 +585,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -744,21 +746,42 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + if (opts_out->delim) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DELIMITER")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + if (opts_out->null_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "NULL")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "NULL")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + if (opts_out->default_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "DEFAULT")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DEFAULT")); + } /* Set defaults for omitted options */ if (!opts_out->delim) @@ -824,11 +847,18 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + if (opts_out->header_line != COPY_HEADER_FALSE) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "HEADER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot specify %s in JSON mode", "HEADER")); + } /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -932,6 +962,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 880231ea6cd..f3f257eaef2 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -130,6 +132,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +152,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV and text, json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +319,46 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + StringInfo result; + + /* + * If COPY TO source data come from query rather than plain table, we need + * copy CopyToState->QueryDesc->TupleDesc to slot->tts_tupleDescriptor. + * This is necessary because the slot's TupleDesc may change during query + * execution, and we depend on it when calling composite_to_json. + */ + if (!cstate->rel) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + } + rowdata = ExecFetchSlotHeapTupleDatum(slot); + result = makeStringInfo(); + composite_to_json(rowdata, result, false); + + CopySendData(cstate, result->data, result->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +450,21 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * JSON format is always one non-binary column + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -504,7 +564,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 713ee5c10a2..f70a465bb64 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3609,6 +3609,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3691,6 +3695,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 78e84727fdc..eec6c5f5ea1 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 8b91bc00062..37cb64d2b72 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3413,7 +3413,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 7be41d9da12..fe85f02b110 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -56,6 +56,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 24e0f472f14..10333357d68 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,82 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY json mode cannot be used with COPY FROM +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 676a8b342b5..80799e2ead9 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,53 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-02-03 16:40 Florents Tselai <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Florents Tselai @ 2026-02-03 16:40 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Junwang Zhao <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Wed, Jan 21, 2026 at 2:16 PM jian he <[email protected]> wrote: > hi. > > rebase only. > > -- > jian > https://www.enterprisedb.com/ I (and others I assume) would really like to see this in 19; glancing at the thread above and in the test cases I see this is in good shape for comitter review. No? If I were to add something it would be an example in copy.sgml <para> When the <literal>FORCE_ARRAY</literal> option is enabled, the entire output is wrapped in a JSON array and individual rows are separated by commas: <programlisting> COPY (SELECT id, name FROM users) TO STDOUT (FORMAT JSON, FORCE_ARRAY); </programlisting> <programlisting> [ {"id": 1, "name": "Alice"} ,{"id": 2, "name": "Bob"} ,{"id": 3, "name": "Charlie"} ] </programlisting> </para> Also, apologies if that has been discussed already, is there a good reason why didn't we just go with a simple "WRAP_ARRAY" ? ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-02-06 03:26 jian he <[email protected]> parent: Florents Tselai <[email protected]> 0 siblings, 2 replies; 22+ messages in thread From: jian he @ 2026-02-06 03:26 UTC (permalink / raw) To: Florents Tselai <[email protected]>; +Cc: Junwang Zhao <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Wed, Feb 4, 2026 at 12:41 AM Florents Tselai <[email protected]> wrote: > > I (and others I assume) would really like to see this in 19; > glancing at the thread above and in the test cases I see this is in good shape for comitter review. > No? > > If I were to add something it would be an example in copy.sgml > <para> > When the <literal>FORCE_ARRAY</literal> option is enabled, > the entire output is wrapped in a JSON array and individual rows are separated by commas: > <programlisting> > COPY (SELECT id, name FROM users) TO STDOUT (FORMAT JSON, FORCE_ARRAY); > </programlisting> > <programlisting> > [ > {"id": 1, "name": "Alice"} > ,{"id": 2, "name": "Bob"} > ,{"id": 3, "name": "Charlie"} > ] > </programlisting> > </para> > v23-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch I've added: +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + > Also, apologies if that has been discussed already, > is there a good reason why didn't we just go with a simple "WRAP_ARRAY" ? > I don’t have a particular preference. If the consensus is that WRAP_ARRAY is better than FORCE_ARRAY, we can change it accordingly. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v23-0002-json-format-for-COPY-TO.patch (21.9K, 2-v23-0002-json-format-for-COPY-TO.patch) download | inline diff: From 89a0ab3121739b0243fdfb80fc22484c5f1b6c26 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:43:14 +0800 Subject: [PATCH v23 2/3] json format for COPY TO JSON format is only supported with the COPY TO operation. It is incompatible with options such as HEADER, DEFAULT, NULL, DELIMITER, and several others. This has been thoroughly tested in src/test/regress/sql/copy.sql The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues, and further adapted by Junwang Zhao to support the newly introduced CopyToRoutine struct (commit 2e4127b6d2). Author: Joe Conway <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 72 +++++++++++++++++++++------- src/backend/commands/copyto.c | 76 ++++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 ++++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 76 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 47 ++++++++++++++++++ 10 files changed, 268 insertions(+), 34 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 53b0ea8f573..320f5f1edd4 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 88fafcc9949..f83c632a069 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -598,6 +598,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -757,21 +759,42 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + if (opts_out->delim) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DELIMITER")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + if (opts_out->null_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "NULL")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "NULL")); + } - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + if (opts_out->default_print) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in BINARY mode", "DEFAULT")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DEFAULT")); + } /* Set defaults for omitted options */ if (!opts_out->delim) @@ -837,11 +860,18 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + if (opts_out->header_line != COPY_HEADER_FALSE) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "HEADER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot specify %s in JSON mode", "HEADER")); + } /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -945,6 +975,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 0325a16f82a..e1b3364640e 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -130,6 +132,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +152,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV and text, json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +319,46 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + StringInfo result; + + /* + * If COPY TO source data come from query rather than plain table, we need + * copy CopyToState->QueryDesc->TupleDesc to slot->tts_tupleDescriptor. + * This is necessary because the slot's TupleDesc may change during query + * execution, and we depend on it when calling composite_to_json. + */ + if (!cstate->rel) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + } + rowdata = ExecFetchSlotHeapTupleDatum(slot); + result = makeStringInfo(); + composite_to_json(rowdata, result, false); + + CopySendData(cstate, result->data, result->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +450,21 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * JSON format is always one non-binary column + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -506,7 +566,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 713ee5c10a2..f70a465bb64 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3609,6 +3609,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3691,6 +3695,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 8b91bc00062..37cb64d2b72 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3413,7 +3413,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 7be41d9da12..fe85f02b110 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -56,6 +56,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..27a28fee77b 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,82 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY json mode cannot be used with COPY FROM +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..a54af636317 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,53 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 [text/x-patch] v23-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (10.3K, 3-v23-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 3f5706ca51b5c424d679924521039b8645462179 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Thu, 5 Feb 2026 16:37:29 +0800 Subject: [PATCH v23 3/3] Add option force_array for COPY JSON FORMAT force_array option can only be used in COPY TO with JSON format. it make the output json output behave like json array type. refactored by Junwang Zhao to adapt the newly introduced CopyToRoutine struct(2e4127b6d2). Author: Joe Conway <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++++ src/backend/commands/copy.c | 13 +++++++++++ src/backend/commands/copyto.c | 37 +++++++++++++++++++++++++++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 +++++++++++++++++++ src/test/regress/sql/copy.sql | 8 +++++++ 7 files changed, 112 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 320f5f1edd4..afea9caf5ad 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1091,6 +1105,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index f83c632a069..5d9b08b822c 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -570,6 +570,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -726,6 +727,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -981,6 +989,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index e1b3364640e..c9f8f4721d4 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,10 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* need delimiter to start next json array element */ + bool json_row_delim_needed; + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -133,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +182,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +248,16 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If JSON has been requested, and FORCE_ARRAY has been specified send the + * opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -354,11 +369,31 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) result = makeStringInfo(); composite_to_json(rowdata, result, false); + if (cstate->json_row_delim_needed && cstate->opts.force_array) + CopySendChar(cstate, ','); + else if (cstate->opts.force_array) + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + CopySendData(cstate, result->data, result->len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 37cb64d2b72..a610ef603f4 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index fe85f02b110..b2a68fa6407 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,6 +87,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 27a28fee77b..86532a21eed 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -112,6 +112,29 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); ERROR: COPY json mode cannot be used with COPY FROM -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only used with JSON mode +--ok +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index a54af636317..6bc8c4f01d6 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -101,6 +101,14 @@ copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); + +--ok +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v23-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 4-v23-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From 9ea6790293fba0f907e92d188c91612b7eea6164 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v23 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 155a79a70c5..88fafcc9949 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -577,6 +577,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -591,11 +593,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -755,31 +757,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -827,7 +829,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -835,43 +837,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -885,8 +887,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -901,8 +903,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -926,7 +928,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -962,7 +964,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -979,7 +981,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 25ee20b23db..0c73b0259c3 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -155,9 +155,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -261,7 +261,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 94d6f415a06..0d9199f5ce8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -171,7 +171,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -749,7 +749,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -776,7 +776,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9ceeff6d99e..0325a16f82a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 43c2580539f..7be41d9da12 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -48,6 +48,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -58,9 +68,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..d061e55d1e8 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -527,6 +527,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-02-06 06:19 Junwang Zhao <[email protected]> parent: jian he <[email protected]> 1 sibling, 0 replies; 22+ messages in thread From: Junwang Zhao @ 2026-02-06 06:19 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Florents Tselai <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Fri, Feb 6, 2026 at 11:26 AM jian he <[email protected]> wrote: > > On Wed, Feb 4, 2026 at 12:41 AM Florents Tselai > <[email protected]> wrote: > > > > I (and others I assume) would really like to see this in 19; > > glancing at the thread above and in the test cases I see this is in good shape for comitter review. > > No? > > > > If I were to add something it would be an example in copy.sgml > > <para> > > When the <literal>FORCE_ARRAY</literal> option is enabled, > > the entire output is wrapped in a JSON array and individual rows are separated by commas: > > <programlisting> > > COPY (SELECT id, name FROM users) TO STDOUT (FORMAT JSON, FORCE_ARRAY); > > </programlisting> > > <programlisting> > > [ > > {"id": 1, "name": "Alice"} > > ,{"id": 2, "name": "Bob"} > > ,{"id": 3, "name": "Charlie"} > > ] > > </programlisting> > > </para> > > > > v23-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch > I've added: > > +<para> > + When the <literal>FORCE_ARRAY</literal> option is enabled, > + the entire output is wrapped in a single JSON array with rows > separated by commas: > +<programlisting> > +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, > FORCE_ARRAY); > +</programlisting> > +The output is as follows: > +<screen> > +[ > + {"id":1} > +,{"id":2} > +] > +</screen> > +</para> > + > + > > > Also, apologies if that has been discussed already, > > is there a good reason why didn't we just go with a simple "WRAP_ARRAY" ? > > > > I don’t have a particular preference. > If the consensus is that WRAP_ARRAY is better than FORCE_ARRAY, we can > change it accordingly. +0.5 for WRAP_ARRAY > > > -- > jian > https://www.enterprisedb.com/ -- Regards Junwang Zhao ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-02-07 13:27 Junwang Zhao <[email protected]> parent: jian he <[email protected]> 1 sibling, 1 reply; 22+ messages in thread From: Junwang Zhao @ 2026-02-07 13:27 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Florents Tselai <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> Hi jian, Thanks for keeping the patch set up to date with the master. On Fri, Feb 6, 2026 at 11:26 AM jian he <[email protected]> wrote: > > On Wed, Feb 4, 2026 at 12:41 AM Florents Tselai > <[email protected]> wrote: > > > > I (and others I assume) would really like to see this in 19; > > glancing at the thread above and in the test cases I see this is in good shape for comitter review. > > No? > > > > If I were to add something it would be an example in copy.sgml > > <para> > > When the <literal>FORCE_ARRAY</literal> option is enabled, > > the entire output is wrapped in a JSON array and individual rows are separated by commas: > > <programlisting> > > COPY (SELECT id, name FROM users) TO STDOUT (FORMAT JSON, FORCE_ARRAY); > > </programlisting> > > <programlisting> > > [ > > {"id": 1, "name": "Alice"} > > ,{"id": 2, "name": "Bob"} > > ,{"id": 3, "name": "Charlie"} > > ] > > </programlisting> > > </para> > > > > v23-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch > I've added: > > +<para> > + When the <literal>FORCE_ARRAY</literal> option is enabled, > + the entire output is wrapped in a single JSON array with rows > separated by commas: > +<programlisting> > +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, > FORCE_ARRAY); > +</programlisting> > +The output is as follows: > +<screen> > +[ > + {"id":1} > +,{"id":2} > +] > +</screen> > +</para> > + > + > > > Also, apologies if that has been discussed already, > > is there a good reason why didn't we just go with a simple "WRAP_ARRAY" ? > > > > I don’t have a particular preference. > If the consensus is that WRAP_ARRAY is better than FORCE_ARRAY, we can > change it accordingly. > > > -- > jian > https://www.enterprisedb.com/ Here are some comments on v23: 0001: The refactor looks straightforward to me. Introducing a format field should make future extensions easier. One suggestion is that we could add some helper macros around format, for example: #define IS_FORMAT_CSV(format) (format == COPY_FORMAT_CSV) #define IS_FORMAT_TEXT_LIKE(format) \ (format == COPY_FORMAT_TEXT || format == COPY_FORMAT_CSV) I think this would improve readability. 0002: Since you have moved the `CopyFormat enum` into 0001, the following commit msg should be rephrased. The CopyFormat enum was originally contributed by Joel Jacobson [email protected], later refactored by Jian He to address various issues, and further adapted by Junwang Zhao to support the newly introduced CopyToRoutine struct (commit 2e4127b6d2). - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) - ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + if (opts_out->delim) + { + if (opts_out->format == COPY_FORMAT_BINARY) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); + else if (opts_out->format == COPY_FORMAT_JSON) + ereport(ERROR, + errcode(ERRCODE_SYNTAX_ERROR), + errmsg("cannot specify %s in JSON mode", "DELIMITER")); + } Can we add a function that converts CopyFormat to a string? Treating CopyFormat as %s in error messages would make the code shorter. However, I'm not sure whether this aligns with translation conventions, correct me if I'm wrong. - * CSV and text formats share the same TextLike routines except for the + * CSV and text, json formats share the same TextLike routines except for the I'd suggest rewording to `CSV, text and json ...`. The same applied to other parts in this patch. 0003: The commit message includes some changes(adapt the newly introduced CopyToRoutine) that actually belong to 0002; it would be better to remove them from this commit. + if (cstate->json_row_delim_needed && cstate->opts.force_array) + CopySendChar(cstate, ','); + else if (cstate->opts.force_array) + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } can we do this: if (cstate->opts.force_array) { if (cstate->json_row_delim_needed) CopySendChar(cstate, ','); else { /* first row needs no delimiter */ CopySendChar(cstate, ' '); cstate->json_row_delim_needed = true; } } -- Regards Junwang Zhao ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-02-09 03:48 jian he <[email protected]> parent: Junwang Zhao <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2026-02-09 03:48 UTC (permalink / raw) To: Junwang Zhao <[email protected]>; +Cc: Florents Tselai <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Andrew Dunstan <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Sat, Feb 7, 2026 at 9:27 PM Junwang Zhao <[email protected]> wrote: > > Here are some comments on v23: > > 0001: The refactor looks straightforward to me. Introducing a format > field should make future extensions easier. One suggestion is that we > could add some helper macros around format, for example: > > #define IS_FORMAT_CSV(format) (format == COPY_FORMAT_CSV) > #define IS_FORMAT_TEXT_LIKE(format) \ > (format == COPY_FORMAT_TEXT || format == COPY_FORMAT_CSV) > > I think this would improve readability. Personally, I don't like marcos.... > > 0002: Since you have moved the `CopyFormat enum` into 0001, the > following commit msg should be rephrased. > > The CopyFormat enum was originally contributed by Joel Jacobson > [email protected], later refactored by Jian He to address various issues, and > further adapted by Junwang Zhao to support the newly introduced CopyToRoutine > struct (commit 2e4127b6d2). > > - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) > - ereport(ERROR, > - (errcode(ERRCODE_SYNTAX_ERROR), > - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ > - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); > + if (opts_out->delim) > + { > + if (opts_out->format == COPY_FORMAT_BINARY) > + ereport(ERROR, > + errcode(ERRCODE_SYNTAX_ERROR), > + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ > + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); > + else if (opts_out->format == COPY_FORMAT_JSON) > + ereport(ERROR, > + errcode(ERRCODE_SYNTAX_ERROR), > + errmsg("cannot specify %s in JSON mode", "DELIMITER")); > + } > > Can we add a function that converts CopyFormat to a string? Treating > CopyFormat as %s in error messages would make the code shorter. > However, I'm not sure whether this aligns with translation > conventions, correct me if I'm wrong. > I don’t think this is worth the added complexity. That said, I tried to simplify the code and changed it to: if (opts_out->delim && (opts_out->format == COPY_FORMAT_BINARY || opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, errcode(ERRCODE_SYNTAX_ERROR), opts_out->format == COPY_FORMAT_BINARY ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") : errmsg("cannot specify %s in JSON mode", "DELIMITER")); > - * CSV and text formats share the same TextLike routines except for the > + * CSV and text, json formats share the same TextLike routines except for the > > I'd suggest rewording to `CSV, text and json ...`. The same applied to > other parts in this patch. > sure. > 0003: The commit message includes some changes(adapt the newly > introduced CopyToRoutine) that actually belong to 0002; it would be > better to remove them from this commit. > 0002 commit message: """ This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. """ 0003 commit message: """ Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. """ > + if (cstate->json_row_delim_needed && cstate->opts.force_array) > + CopySendChar(cstate, ','); > + else if (cstate->opts.force_array) > + { > + /* first row needs no delimiter */ > + CopySendChar(cstate, ' '); > + cstate->json_row_delim_needed = true; > + } > > can we do this: > > if (cstate->opts.force_array) > { > if (cstate->json_row_delim_needed) > CopySendChar(cstate, ','); > else > { > /* first row needs no delimiter */ > CopySendChar(cstate, ' '); > cstate->json_row_delim_needed = true; > } > } > good suggestion. If more people think WRAP_ARRAY is better than FORCE_ARRAY, we can switch to it accordingly. The change itself is quite straightforward. -- jian https://www.enterprisedb.com Attachments: [text/x-patch] v24-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 2-v24-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From 1cde2b1b7d5a8e9150d90ecf4807da7dca934dad Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v24 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 155a79a70c5..88fafcc9949 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -577,6 +577,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -591,11 +593,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -755,31 +757,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -827,7 +829,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -835,43 +837,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -885,8 +887,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -901,8 +903,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -926,7 +928,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -962,7 +964,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -979,7 +981,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 25ee20b23db..0c73b0259c3 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -155,9 +155,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -261,7 +261,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 94d6f415a06..0d9199f5ce8 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -171,7 +171,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -749,7 +749,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -776,7 +776,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9ceeff6d99e..0325a16f82a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 43c2580539f..7be41d9da12 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -48,6 +48,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -58,9 +68,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 9f5ee8fd482..d061e55d1e8 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -527,6 +527,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 [text/x-patch] v24-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (10.7K, 3-v24-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From d6a4166fd5d6899895b223594a9497adcdff3f51 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Feb 2026 11:14:15 +0800 Subject: [PATCH v24 3/3] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: jian he <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 40 +++++++++++++++++++++++++++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 +++++++++++++++++ src/test/regress/sql/copy.sql | 8 ++++++ 7 files changed, 115 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 320f5f1edd4..afea9caf5ad 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1091,6 +1105,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index f16ca988126..841a84848cd 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -570,6 +570,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -726,6 +727,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -968,6 +976,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 720ce0267bf..33e7c30c761 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,10 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* need delimiter to start next json array element */ + bool json_row_delim_needed; + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -133,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +182,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +248,16 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If JSON has been requested, and FORCE_ARRAY has been specified send the + * opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -354,11 +369,34 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) result = makeStringInfo(); composite_to_json(rowdata, result, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, result->data, result->len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 37cb64d2b72..a610ef603f4 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index fe85f02b110..b2a68fa6407 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,6 +87,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 27a28fee77b..86532a21eed 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -112,6 +112,29 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); ERROR: COPY json mode cannot be used with COPY FROM -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only used with JSON mode +--ok +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index a54af636317..6bc8c4f01d6 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -101,6 +101,14 @@ copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -- all of the above should yield error +--Error +copy copytest to stdout (format csv, force_array true); + +--ok +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v24-0002-json-format-for-COPY-TO.patch (21.8K, 4-v24-0002-json-format-for-COPY-TO.patch) download | inline diff: From 3f878129819478be3ecead3cffd64f28de95775e Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Feb 2026 11:00:06 +0800 Subject: [PATCH v24 2/3] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 49 ++++++++++++++----- src/backend/commands/copyto.c | 76 ++++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 ++++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 76 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 47 ++++++++++++++++++ 10 files changed, 250 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 53b0ea8f573..320f5f1edd4 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 88fafcc9949..f16ca988126 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -598,6 +598,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -757,21 +759,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -837,11 +850,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -945,6 +962,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "json", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 0325a16f82a..720ce0267bf 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -130,6 +132,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +152,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV, text and json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +319,46 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + StringInfo result; + + /* + * If COPY TO source data come from query rather than plain table, we need + * copy CopyToState->QueryDesc->TupleDesc to slot->tts_tupleDescriptor. + * This is necessary because the slot's TupleDesc may change during query + * execution, and we depend on it when calling composite_to_json. + */ + if (!cstate->rel) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + } + rowdata = ExecFetchSlotHeapTupleDatum(slot); + result = makeStringInfo(); + composite_to_json(rowdata, result, false); + + CopySendData(cstate, result->data, result->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +450,21 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * JSON format is always one non-binary column + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -506,7 +566,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 713ee5c10a2..f70a465bb64 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3609,6 +3609,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3691,6 +3695,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 8b91bc00062..37cb64d2b72 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3413,7 +3413,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 7be41d9da12..fe85f02b110 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -56,6 +56,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..27a28fee77b 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,82 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY json mode cannot be used with COPY FROM +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..a54af636317 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,53 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-04 15:51 Andrew Dunstan <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Andrew Dunstan @ 2026-03-04 15:51 UTC (permalink / raw) To: jian he <[email protected]>; Junwang Zhao <[email protected]>; +Cc: Florents Tselai <[email protected]>; Joe Conway <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-02-08 Su 10:48 PM, jian he wrote: > On Sat, Feb 7, 2026 at 9:27 PM Junwang Zhao<[email protected]> wrote: >> Here are some comments on v23: >> >> 0001: The refactor looks straightforward to me. Introducing a format >> field should make future extensions easier. One suggestion is that we >> could add some helper macros around format, for example: >> >> #define IS_FORMAT_CSV(format) (format == COPY_FORMAT_CSV) >> #define IS_FORMAT_TEXT_LIKE(format) \ >> (format == COPY_FORMAT_TEXT || format == COPY_FORMAT_CSV) >> >> I think this would improve readability. > Personally, I don't like marcos.... > >> 0002: Since you have moved the `CopyFormat enum` into 0001, the >> following commit msg should be rephrased. >> >> The CopyFormat enum was originally contributed by Joel Jacobson >> [email protected], later refactored by Jian He to address various issues, and >> further adapted by Junwang Zhao to support the newly introduced CopyToRoutine >> struct (commit 2e4127b6d2). >> >> - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) >> - ereport(ERROR, >> - (errcode(ERRCODE_SYNTAX_ERROR), >> - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ >> - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); >> + if (opts_out->delim) >> + { >> + if (opts_out->format == COPY_FORMAT_BINARY) >> + ereport(ERROR, >> + errcode(ERRCODE_SYNTAX_ERROR), >> + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ >> + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); >> + else if (opts_out->format == COPY_FORMAT_JSON) >> + ereport(ERROR, >> + errcode(ERRCODE_SYNTAX_ERROR), >> + errmsg("cannot specify %s in JSON mode", "DELIMITER")); >> + } >> >> Can we add a function that converts CopyFormat to a string? Treating >> CopyFormat as %s in error messages would make the code shorter. >> However, I'm not sure whether this aligns with translation >> conventions, correct me if I'm wrong. >> > I don’t think this is worth the added complexity. > That said, I tried to simplify the code and changed it to: > > if (opts_out->delim && > (opts_out->format == COPY_FORMAT_BINARY || > opts_out->format == COPY_FORMAT_JSON)) > ereport(ERROR, > errcode(ERRCODE_SYNTAX_ERROR), > opts_out->format == COPY_FORMAT_BINARY > ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") > : errmsg("cannot specify %s in JSON mode", "DELIMITER")); > >> - * CSV and text formats share the same TextLike routines except for the >> + * CSV and text, json formats share the same TextLike routines except for the >> >> I'd suggest rewording to `CSV, text and json ...`. The same applied to >> other parts in this patch. >> > sure. > >> 0003: The commit message includes some changes(adapt the newly >> introduced CopyToRoutine) that actually belong to 0002; it would be >> better to remove them from this commit. >> > 0002 commit message: > """ > This introduces the JSON format option for the COPY TO command, allowing users > to export query results or table data directly as a single JSON object or a > stream of JSON objects. > > The JSON format is currently supported only for COPY TO operations; it > is not available for COPY FROM. > > JSON format is incompatible with some standard text/CSV parsing or > formatting options, > including: > - HEADER > - DEFAULT > - NULL > - DELIMITER > - FORCE QUOTE / FORCE NOT NULL > > Regression tests covering valid JSON exports and error handling for > incompatible options have been added to src/test/regress/sql/copy.sql. > """ > > 0003 commit message: > """ > Add option force_array for COPY JSON FORMAT > This adds the force_array option, which is available exclusively > when using COPY TO with the JSON format. > > When enabled, this option wraps the output in a top-level JSON array > (enclosed in square brackets with comma-separated elements), making the > entire result a valid single JSON value. Without this option, the default > behavior is to output a stream of independent JSON objects. > > Attempting to use this option with COPY FROM or with formats other than > JSON will raise an error. > """ > >> + if (cstate->json_row_delim_needed && cstate->opts.force_array) >> + CopySendChar(cstate, ','); >> + else if (cstate->opts.force_array) >> + { >> + /* first row needs no delimiter */ >> + CopySendChar(cstate, ' '); >> + cstate->json_row_delim_needed = true; >> + } >> >> can we do this: >> >> if (cstate->opts.force_array) >> { >> if (cstate->json_row_delim_needed) >> CopySendChar(cstate, ','); >> else >> { >> /* first row needs no delimiter */ >> CopySendChar(cstate, ' '); >> cstate->json_row_delim_needed = true; >> } >> } >> > good suggestion. > > If more people think WRAP_ARRAY is better than FORCE_ARRAY, we can > switch to it accordingly. > The change itself is quite straightforward. I have reworked these. First I cleaned up a number of things in patches 2 and 3 (Thanks, Claude for the summary): Patch 2: json format for COPY TO copy.c: - "json" → "JSON" in the COPY FROM rejection error message. copyto.c: 1. TupleDesc setup runs once, not every row — Added json_tupledesc_ready flag; the memcpy/populate_compact_attribute/BlessTupleDesc block is now guarded by if (!cstate->json_tupledesc_ready). 2. Comment rewritten — Old: "the slot's TupleDesc may change during query execution". New: explains BlessTupleDesc registers the RECORDOID descriptor so lookup_rowtype_tupdesc inside composite_to_json can find it. 3. Eliminated per-row makeStringInfo() — Added StringInfoData json_buf to the struct, initialized once in CopyToTextLikeStart in copycontext. Each row does resetStringInfo instead of allocating a new StringInfo. 4. Column list rejection added — Error: "column selection is not supported in JSON mode" when attnamelist != NIL. 5. Improved SendCopyBegin comment — Old: "JSON format is always one non-binary column". New: explains each CopyData message contains one complete JSON object. Tests: 6. Added copy copytest (style) to stdout (format json) to the error-case block. 7. Added copyjsontype table test with json, jsonb columns — verifies values are embedded directly, not double-encoded. Covers json objects, scalars, arrays, nested objects, and nulls. Patch 3: Add option force_array copy.c: 1. "json" → "JSON" in COPY FROM error (carried from patch 2). 2. "can only used" → "can only be used" — grammar fix in FORCE_ARRAY error. copyto.c: 3. Struct fields reorganized — JSON fields grouped under /* JSON format state */ comment with inline descriptions, instead of a standalone json_row_delim_needed with a vague comment. 4. Block comment updated — Was "CSV, text and json formats share the same TextLike routines except for the one-row callback". Now correctly notes JSON has its own one-row and end callbacks. 5. CopyToTextLikeEnd comment fixed — Was "text, CSV, and json", now "text and CSV" (JSON uses CopyToJsonEnd). 6. Cleaner start callback — FORCE_ARRAY bracket emission moved inside the if (format == JSON) block after json_buf init, instead of a separate top-level conditional. 7. Inherits all patch 2 fixes — TupleDesc guard, reusable json_buf, improved comments, column list rejection. Tests: 8. --Error → -- should fail: force_array requires json format; --ok → -- force_array variants. 9. copyjsontype test carried through from patch 2. Then I reworked the way this works. In order to support column lists with JSON output, we need to deal with individual columns instead of whole records. This involved quite a number of changes, as can be seen in patch 4. This involved exporting a new small function from json.c. The result is a lot cleaner, I believe, and in my benchmarking is faster by a factor of almost 2. cheers andrew -- Andrew Dunstan EDB:https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-05 18:06 Joe Conway <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Joe Conway @ 2026-03-05 18:06 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; jian he <[email protected]>; Junwang Zhao <[email protected]>; +Cc: Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 3/4/26 10:51, Andrew Dunstan wrote: > > On 2026-02-08 Su 10:48 PM, jian he wrote: >> On Sat, Feb 7, 2026 at 9:27 PM Junwang Zhao<[email protected]> wrote: >>> Here are some comments on v23: >>> >>> 0001: The refactor looks straightforward to me. Introducing a format >>> field should make future extensions easier. One suggestion is that we >>> could add some helper macros around format, for example: >>> >>> #define IS_FORMAT_CSV(format) (format == COPY_FORMAT_CSV) >>> #define IS_FORMAT_TEXT_LIKE(format) \ >>> (format == COPY_FORMAT_TEXT || format == COPY_FORMAT_CSV) >>> >>> I think this would improve readability. >> Personally, I don't like marcos.... >> >>> 0002: Since you have moved the `CopyFormat enum` into 0001, the >>> following commit msg should be rephrased. >>> >>> The CopyFormat enum was originally contributed by Joel Jacobson >>> [email protected], later refactored by Jian He to address various issues, and >>> further adapted by Junwang Zhao to support the newly introduced CopyToRoutine >>> struct (commit 2e4127b6d2). >>> >>> - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) >>> - ereport(ERROR, >>> - (errcode(ERRCODE_SYNTAX_ERROR), >>> - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ >>> - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); >>> + if (opts_out->delim) >>> + { >>> + if (opts_out->format == COPY_FORMAT_BINARY) >>> + ereport(ERROR, >>> + errcode(ERRCODE_SYNTAX_ERROR), >>> + /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ >>> + errmsg("cannot specify %s in BINARY mode", "DELIMITER")); >>> + else if (opts_out->format == COPY_FORMAT_JSON) >>> + ereport(ERROR, >>> + errcode(ERRCODE_SYNTAX_ERROR), >>> + errmsg("cannot specify %s in JSON mode", "DELIMITER")); >>> + } >>> >>> Can we add a function that converts CopyFormat to a string? Treating >>> CopyFormat as %s in error messages would make the code shorter. >>> However, I'm not sure whether this aligns with translation >>> conventions, correct me if I'm wrong. >>> >> I don’t think this is worth the added complexity. >> That said, I tried to simplify the code and changed it to: >> >> if (opts_out->delim && >> (opts_out->format == COPY_FORMAT_BINARY || >> opts_out->format == COPY_FORMAT_JSON)) >> ereport(ERROR, >> errcode(ERRCODE_SYNTAX_ERROR), >> opts_out->format == COPY_FORMAT_BINARY >> ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") >> : errmsg("cannot specify %s in JSON mode", "DELIMITER")); >> >>> - * CSV and text formats share the same TextLike routines except for the >>> + * CSV and text, json formats share the same TextLike routines except for the >>> >>> I'd suggest rewording to `CSV, text and json ...`. The same applied to >>> other parts in this patch. >>> >> sure. >> >>> 0003: The commit message includes some changes(adapt the newly >>> introduced CopyToRoutine) that actually belong to 0002; it would be >>> better to remove them from this commit. >>> >> 0002 commit message: >> """ >> This introduces the JSON format option for the COPY TO command, allowing users >> to export query results or table data directly as a single JSON object or a >> stream of JSON objects. >> >> The JSON format is currently supported only for COPY TO operations; it >> is not available for COPY FROM. >> >> JSON format is incompatible with some standard text/CSV parsing or >> formatting options, >> including: >> - HEADER >> - DEFAULT >> - NULL >> - DELIMITER >> - FORCE QUOTE / FORCE NOT NULL >> >> Regression tests covering valid JSON exports and error handling for >> incompatible options have been added to src/test/regress/sql/copy.sql. >> """ >> >> 0003 commit message: >> """ >> Add option force_array for COPY JSON FORMAT >> This adds the force_array option, which is available exclusively >> when using COPY TO with the JSON format. >> >> When enabled, this option wraps the output in a top-level JSON array >> (enclosed in square brackets with comma-separated elements), making the >> entire result a valid single JSON value. Without this option, the default >> behavior is to output a stream of independent JSON objects. >> >> Attempting to use this option with COPY FROM or with formats other than >> JSON will raise an error. >> """ >> >>> + if (cstate->json_row_delim_needed && cstate->opts.force_array) >>> + CopySendChar(cstate, ','); >>> + else if (cstate->opts.force_array) >>> + { >>> + /* first row needs no delimiter */ >>> + CopySendChar(cstate, ' '); >>> + cstate->json_row_delim_needed = true; >>> + } >>> >>> can we do this: >>> >>> if (cstate->opts.force_array) >>> { >>> if (cstate->json_row_delim_needed) >>> CopySendChar(cstate, ','); >>> else >>> { >>> /* first row needs no delimiter */ >>> CopySendChar(cstate, ' '); >>> cstate->json_row_delim_needed = true; >>> } >>> } >>> >> good suggestion. >> >> If more people think WRAP_ARRAY is better than FORCE_ARRAY, we can >> switch to it accordingly. >> The change itself is quite straightforward. > > > I have reworked these. First I cleaned up a number of things in patches > 2 and 3 (Thanks, Claude for the summary): > > Patch 2: json format for COPY TO > > copy.c: > - "json" → "JSON" in the COPY FROM rejection error message. > > copyto.c: > 1. TupleDesc setup runs once, not every row — Added > json_tupledesc_ready flag; the memcpy/populate_compact_attribute/ > BlessTupleDesc block is now guarded by if (!cstate->json_tupledesc_ready). > 2. Comment rewritten — Old: "the slot's TupleDesc may change during > query execution". New: explains BlessTupleDesc registers the RECORDOID > descriptor so lookup_rowtype_tupdesc inside composite_to_json can > find it. > 3. Eliminated per-row makeStringInfo() — Added StringInfoData > json_buf to the struct, initialized once in CopyToTextLikeStart in > copycontext. Each row does resetStringInfo instead of allocating a new > StringInfo. > 4. Column list rejection added — Error: "column selection is not > supported in JSON mode" when attnamelist != NIL. > 5. Improved SendCopyBegin comment — Old: "JSON format is always one > non-binary column". New: explains each CopyData message contains one > complete JSON object. > > Tests: > 6. Added copy copytest (style) to stdout (format json) to the error- > case block. > 7. Added copyjsontype table test with json, jsonb columns — verifies > values are embedded directly, not double-encoded. Covers json objects, > scalars, arrays, nested objects, and nulls. > > Patch 3: Add option force_array > > copy.c: > 1. "json" → "JSON" in COPY FROM error (carried from patch 2). > 2. "can only used" → "can only be used" — grammar fix in FORCE_ARRAY > error. > > copyto.c: > 3. Struct fields reorganized — JSON fields grouped under /* JSON > format state */ comment with inline descriptions, instead of a > standalone json_row_delim_needed with a vague comment. > 4. Block comment updated — Was "CSV, text and json formats share the > same TextLike routines except for the one-row callback". Now correctly > notes JSON has its own one-row and end callbacks. > 5. CopyToTextLikeEnd comment fixed — Was "text, CSV, and json", now > "text and CSV" (JSON uses CopyToJsonEnd). > 6. Cleaner start callback — FORCE_ARRAY bracket emission moved inside > the if (format == JSON) block after json_buf init, instead of a separate > top-level conditional. > 7. Inherits all patch 2 fixes — TupleDesc guard, reusable json_buf, > improved comments, column list rejection. > > Tests: > 8. --Error → -- should fail: force_array requires json format; --ok → > -- force_array variants. > 9. copyjsontype test carried through from patch 2. > > Then I reworked the way this works. In order to support column lists > with JSON output, we need to deal with individual columns instead of > whole records. This involved quite a number of changes, as can be seen > in patch 4. This involved exporting a new small function from json.c. > > The result is a lot cleaner, I believe, and in my benchmarking is faster > by a factor of almost 2. Andrew, I don't see the actual patches. Did I miss it somewhere? -- Joe Conway PostgreSQL Contributors Team Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-05 20:49 Andrew Dunstan <[email protected]> parent: Joe Conway <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Andrew Dunstan @ 2026-03-05 20:49 UTC (permalink / raw) To: Joe Conway <[email protected]>; jian he <[email protected]>; Junwang Zhao <[email protected]>; +Cc: Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> > On 2026-03-05 Th 1:06 PM, Joe Conway wrote: >> >> >> Then I reworked the way this works. In order to support column lists >> with JSON output, we need to deal with individual columns instead of >> whole records. This involved quite a number of changes, as can be >> seen in patch 4. This involved exporting a new small function from >> json.c. >> >> The result is a lot cleaner, I believe, and in my benchmarking is >> faster by a factor of almost 2. > > Andrew, > > I don't see the actual patches. Did I miss it somewhere? Nope. Bad hair day apparently. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com Attachments: [text/x-patch] v25-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 2-v25-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From b50c26e9e2a76001315fb3e5000f2d33e254c741 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v25 1/4] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2f42f55e229..4d927410159 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index fbd13353efc..c366874bd95 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -172,7 +172,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -750,7 +750,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -777,7 +777,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9ceeff6d99e..0325a16f82a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 77e3c04144e..8399be97fd5 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.43.0 [text/x-patch] v25-0002-json-format-for-COPY-TO.patch (22.2K, 3-v25-0002-json-format-for-COPY-TO.patch) download | inline diff: From 285be0d7935b5505b6cc1509742f3fc87e19ea09 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Feb 2026 11:00:06 +0800 Subject: [PATCH v25 2/4] json format for COPY TO --- doc/src/sgml/ref/copy.sgml | 13 ++-- src/backend/commands/copy.c | 49 ++++++++++---- src/backend/commands/copyto.c | 101 ++++++++++++++++++++++++++--- src/backend/parser/gram.y | 8 +++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 78 ++++++++++++++++++++++ src/test/regress/sql/copy.sql | 48 ++++++++++++++ 10 files changed, 278 insertions(+), 29 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29c121c7f08 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 0325a16f82a..96605079eeb 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -84,6 +86,11 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + + /* JSON format state */ + bool json_tupledesc_ready; /* TupleDesc setup done for JSON */ + StringInfoData json_buf; /* reusable buffer for JSON output */ + copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -130,6 +137,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,7 +157,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV and text formats share the same TextLike routines except for the + * CSV, text and json formats share the same TextLike routines except for the * one-row callback. */ @@ -169,6 +177,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +201,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +227,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -228,10 +248,21 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* JSON-specific initialization */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + MemoryContext oldcxt; + + /* Allocate reusable JSON output buffer in long-lived context */ + oldcxt = MemoryContextSwitchTo(cstate->copycontext); + initStringInfo(&cstate->json_buf); + MemoryContextSwitchTo(oldcxt); + } } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +335,47 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + /* + * For query-based COPY, copy the query's TupleDesc attributes into the + * slot's TupleDesc once. BlessTupleDesc registers the RECORDOID + * descriptor so that lookup_rowtype_tupdesc inside composite_to_json can + * find it. + */ + if (!cstate->rel && !cstate->json_tupledesc_ready) + { + memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), + TupleDescAttr(cstate->queryDesc->tupDesc, 0), + cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); + + for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) + populate_compact_attribute(slot->tts_tupleDescriptor, i); + + BlessTupleDesc(slot->tts_tupleDescriptor); + cstate->json_tupledesc_ready = true; + } + + rowdata = ExecFetchSlotHeapTupleDatum(slot); + resetStringInfo(&cstate->json_buf); + composite_to_json(rowdata, &cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf.data, cstate->json_buf.len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +467,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -506,7 +585,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -890,6 +969,12 @@ BeginCopyTo(ParseState *pstate, /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* JSON outputs whole rows; a column list doesn't make sense */ + if (cstate->opts.format == COPY_FORMAT_JSON && attnamelist != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column selection is not supported in JSON mode"))); + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index c567252acc4..db98f2d91bf 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3609,6 +3609,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3691,6 +3695,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 905c076763c..d257837b0c5 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..4ea658a45de 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,84 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest from stdin(format json); +ERROR: COPY JSON mode cannot be used with COPY FROM +copy copytest (style) to stdout (format json); +ERROR: column selection is not supported in JSON mode +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..c558eba202a 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,54 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest from stdin(format json); +copy copytest (style) to stdout (format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.43.0 [text/x-patch] v25-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (10.7K, 4-v25-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 312dedc0a3aea1833bb992f096121db4086fd075 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Feb 2026 11:14:15 +0800 Subject: [PATCH v25 3/4] Add option force_array for COPY JSON FORMAT --- doc/src/sgml/ref/copy.sgml | 30 +++++++++++++++++++++ src/backend/commands/copy.c | 13 +++++++++ src/backend/commands/copyto.c | 43 +++++++++++++++++++++++++++--- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 23 ++++++++++++++++ src/test/regress/sql/copy.sql | 8 ++++++ 7 files changed, 115 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29c121c7f08..84254d46a67 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 96605079eeb..a7615cc34ec 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -88,6 +88,7 @@ typedef struct CopyToStateData bool is_program; /* is 'filename' a program to popen? */ /* JSON format state */ + bool json_row_delim_needed; /* need delimiter before next row */ bool json_tupledesc_ready; /* TupleDesc setup done for JSON */ StringInfoData json_buf; /* reusable buffer for JSON output */ @@ -138,6 +139,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -157,8 +159,9 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. * - * CSV, text and json formats share the same TextLike routines except for the - * one-row callback. + * Text and CSV formats share the same TextLike routines except for the + * one-row callback. JSON shares the start and outfunc callbacks with + * text/CSV, but has its own one-row and end callbacks. */ /* text format */ @@ -182,7 +185,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -258,6 +261,15 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) oldcxt = MemoryContextSwitchTo(cstate->copycontext); initStringInfo(&cstate->json_buf); MemoryContextSwitchTo(oldcxt); + + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } } @@ -335,7 +347,7 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { @@ -371,11 +383,34 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) resetStringInfo(&cstate->json_buf); composite_to_json(rowdata, &cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf.data, cstate->json_buf.len); CopySendTextLikeEndOfRow(cstate); } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index d257837b0c5..76d3258e92b 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 4ea658a45de..a7e88b711d7 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -114,6 +114,29 @@ ERROR: COPY JSON mode cannot be used with COPY FROM copy copytest (style) to stdout (format json); ERROR: column selection is not supported in JSON mode -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index c558eba202a..ae202fc5e8d 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -102,6 +102,14 @@ copy copytest from stdin(format json); copy copytest (style) to stdout (format json); -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.43.0 [text/x-patch] v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch (13.3K, 5-v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch) download | inline diff: From d043adcbb2032eb2c1218df99992008838f2e200 Mon Sep 17 00:00:00 2001 From: Andrew Dunstan <[email protected]> Date: Wed, 4 Mar 2026 09:40:10 -0500 Subject: [PATCH v25 4/4] COPY TO JSON: build JSON per-column, support column lists Rework CopyToJsonOneRow to iterate attnumlist and build JSON objects column-by-column using datum_to_json_append, instead of converting the whole row via ExecFetchSlotHeapTupleDatum + composite_to_json. This has several benefits: - Column lists now work with JSON format (previously rejected) - Per-column JSON type categorization is done once at startup rather than on every row (composite_to_json called json_categorize_type per column per row) - The TupleDesc memcpy/BlessTupleDesc hack for query-based COPY is eliminated entirely - Pre-escaped column key strings avoid repeated escape_json calls Add CopyToJsonStart (pre-computes escaped key strings and json_buf), CopyToJsonOutFunc (calls json_categorize_type once per column), and export datum_to_json_append from json.c for efficient append-to- StringInfo JSON serialization. --- src/backend/commands/copyto.c | 159 ++++++++++++++++++----------- src/backend/utils/adt/json.c | 14 +++ src/include/utils/jsonfuncs.h | 2 + src/test/regress/expected/copy.out | 8 +- src/test/regress/sql/copy.sql | 4 +- 5 files changed, 126 insertions(+), 61 deletions(-) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index a7615cc34ec..9502c910b43 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,7 +26,6 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" -#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -35,6 +34,7 @@ #include "storage/fd.h" #include "tcop/tcopprot.h" #include "utils/json.h" +#include "utils/jsonfuncs.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -89,8 +89,10 @@ typedef struct CopyToStateData /* JSON format state */ bool json_row_delim_needed; /* need delimiter before next row */ - bool json_tupledesc_ready; /* TupleDesc setup done for JSON */ StringInfoData json_buf; /* reusable buffer for JSON output */ + JsonTypeCategory *json_categories; /* per-column JSON type categories */ + Oid *json_outfuncoids; /* per-column JSON output func OIDs */ + char **json_col_keys; /* per-column pre-escaped "key": strings */ copy_data_dest_cb data_dest_cb; /* function for writing data */ @@ -138,6 +140,8 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonStart(CopyToState cstate, TupleDesc tupDesc); +static void CopyToJsonOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); @@ -160,8 +164,8 @@ static void CopySendInt16(CopyToState cstate, int16 val); * COPY TO routines for built-in formats. * * Text and CSV formats share the same TextLike routines except for the - * one-row callback. JSON shares the start and outfunc callbacks with - * text/CSV, but has its own one-row and end callbacks. + * one-row callback. JSON has its own start, outfunc, one-row, and end + * callbacks. */ /* text format */ @@ -182,8 +186,8 @@ static const CopyToRoutine CopyToRoutineCSV = { /* json format */ static const CopyToRoutine CopyToRoutineJson = { - .CopyToStart = CopyToTextLikeStart, - .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToStart = CopyToJsonStart, + .CopyToOutFunc = CopyToJsonOutFunc, .CopyToOneRow = CopyToJsonOneRow, .CopyToEnd = CopyToJsonEnd, }; @@ -211,7 +215,7 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineText; } -/* Implementation of the start callback for text, CSV, and json formats */ +/* Implementation of the start callback for text and CSV formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -230,8 +234,6 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; - Assert(cstate->opts.format != COPY_FORMAT_JSON); - foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -251,30 +253,10 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } - - /* JSON-specific initialization */ - if (cstate->opts.format == COPY_FORMAT_JSON) - { - MemoryContext oldcxt; - - /* Allocate reusable JSON output buffer in long-lived context */ - oldcxt = MemoryContextSwitchTo(cstate->copycontext); - initStringInfo(&cstate->json_buf); - MemoryContextSwitchTo(oldcxt); - - /* - * If FORCE_ARRAY has been specified, send the opening bracket. - */ - if (cstate->opts.force_array) - { - CopySendChar(cstate, '['); - CopySendTextLikeEndOfRow(cstate); - } - } } /* - * Implementation of the outfunc callback for text, CSV, and json formats. Assign + * Implementation of the outfunc callback for text and CSV formats. Assign * the output function data to the given *finfo. */ static void @@ -354,34 +336,95 @@ CopyToTextLikeEnd(CopyToState cstate) /* Nothing to do here */ } +/* + * Implementation of the start callback for json format. + * + * Pre-compute the escaped JSON key strings ('"colname":') for each selected + * column so CopyToJsonOneRow only needs to copy them per row. + */ +static void +CopyToJsonStart(CopyToState cstate, TupleDesc tupDesc) +{ + MemoryContext oldcxt; + StringInfoData keybuf; + + oldcxt = MemoryContextSwitchTo(cstate->copycontext); + + /* Allocate reusable JSON output buffer */ + initStringInfo(&cstate->json_buf); + + /* Pre-build escaped key strings: "\"colname\":" */ + cstate->json_col_keys = palloc0(tupDesc->natts * sizeof(char *)); + initStringInfo(&keybuf); + foreach_int(attnum, cstate->attnumlist) + { + char *colname; + + colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); + + resetStringInfo(&keybuf); + escape_json(&keybuf, colname); + appendStringInfoChar(&keybuf, ':'); + + cstate->json_col_keys[attnum - 1] = pstrdup(keybuf.data); + } + pfree(keybuf.data); + + MemoryContextSwitchTo(oldcxt); + + /* If FORCE_ARRAY, send the opening bracket */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } +} + +/* + * Implementation of the outfunc callback for json format. + * + * Instead of text output functions, we categorize each column's type for + * JSON serialization once so CopyToJsonOneRow can use datum_to_json_append + * directly. + */ +static void +CopyToJsonOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo) +{ + int attidx = finfo - cstate->out_functions; + + json_categorize_type(atttypid, false, + &cstate->json_categories[attidx], + &cstate->json_outfuncoids[attidx]); +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) { - Datum rowdata; + bool needsep = false; - /* - * For query-based COPY, copy the query's TupleDesc attributes into the - * slot's TupleDesc once. BlessTupleDesc registers the RECORDOID - * descriptor so that lookup_rowtype_tupdesc inside composite_to_json can - * find it. - */ - if (!cstate->rel && !cstate->json_tupledesc_ready) - { - memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0), - TupleDescAttr(cstate->queryDesc->tupDesc, 0), - cstate->queryDesc->tupDesc->natts * sizeof(FormData_pg_attribute)); - - for (int i = 0; i < cstate->queryDesc->tupDesc->natts; i++) - populate_compact_attribute(slot->tts_tupleDescriptor, i); - - BlessTupleDesc(slot->tts_tupleDescriptor); - cstate->json_tupledesc_ready = true; - } - - rowdata = ExecFetchSlotHeapTupleDatum(slot); resetStringInfo(&cstate->json_buf); - composite_to_json(rowdata, &cstate->json_buf, false); + appendStringInfoChar(&cstate->json_buf, '{'); + + foreach_int(attnum, cstate->attnumlist) + { + Datum value = slot->tts_values[attnum - 1]; + bool isnull = slot->tts_isnull[attnum - 1]; + + if (needsep) + appendStringInfoChar(&cstate->json_buf, ','); + needsep = true; + + /* Append pre-escaped "key": */ + appendStringInfoString(&cstate->json_buf, + cstate->json_col_keys[attnum - 1]); + + datum_to_json_append(value, isnull, &cstate->json_buf, + cstate->json_categories[attnum - 1], + cstate->json_outfuncoids[attnum - 1]); + } + + appendStringInfoChar(&cstate->json_buf, '}'); if (cstate->opts.force_array) { @@ -1004,12 +1047,6 @@ BeginCopyTo(ParseState *pstate, /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); - /* JSON outputs whole rows; a column list doesn't make sense */ - if (cstate->opts.format == COPY_FORMAT_JSON && attnamelist != NIL) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column selection is not supported in JSON mode"))); - num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ @@ -1204,6 +1241,12 @@ DoCopyTo(CopyToState cstate) /* Get info about the columns we need to process. */ cstate->out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo)); + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* JSON outfunc callback stores per-column type categorization here */ + cstate->json_categories = palloc0(num_phys_attrs * sizeof(JsonTypeCategory)); + cstate->json_outfuncoids = palloc0(num_phys_attrs * sizeof(Oid)); + } foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index f609d7b9417..de81160a831 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -771,6 +771,20 @@ datum_to_json(Datum val, JsonTypeCategory tcategory, Oid outfuncoid) return PointerGetDatum(cstring_to_text_with_len(result.data, result.len)); } +/* + * Append JSON representation of a Datum to a StringInfo. + * + * tcategory and outfuncoid are from a previous call to json_categorize_type. + * If is_null is true, appends "null" regardless of tcategory/outfuncoid. + */ +void +datum_to_json_append(Datum val, bool is_null, StringInfo result, + JsonTypeCategory tcategory, Oid outfuncoid) +{ + datum_to_json_internal(val, is_null, result, tcategory, outfuncoid, + false); +} + /* * json_agg transition function * diff --git a/src/include/utils/jsonfuncs.h b/src/include/utils/jsonfuncs.h index 636f0f55840..12a01451fbb 100644 --- a/src/include/utils/jsonfuncs.h +++ b/src/include/utils/jsonfuncs.h @@ -85,6 +85,8 @@ extern void json_categorize_type(Oid typoid, bool is_jsonb, JsonTypeCategory *tcategory, Oid *outfuncoid); extern Datum datum_to_json(Datum val, JsonTypeCategory tcategory, Oid outfuncoid); +extern void datum_to_json_append(Datum val, bool is_null, StringInfo result, + JsonTypeCategory tcategory, Oid outfuncoid); extern Datum datum_to_jsonb(Datum val, JsonTypeCategory tcategory, Oid outfuncoid); extern Datum jsonb_from_text(text *js, bool unique_keys); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index a7e88b711d7..d60e5a4d32a 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -111,8 +111,6 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); ^ copy copytest from stdin(format json); ERROR: COPY JSON mode cannot be used with COPY FROM -copy copytest (style) to stdout (format json); -ERROR: column selection is not supported in JSON mode -- all of the above should yield error -- should fail: force_array requires json format copy copytest to stdout (format csv, force_array true); @@ -137,6 +135,12 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with json format +copy copytest (style, filler) to stdout (format json); +{"style":"DOS","filler":1} +{"style":"Unix","filler":2} +{"style":"Mac","filler":3} +{"style":"esc\\ape","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index ae202fc5e8d..d64f4c66b93 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -99,7 +99,6 @@ copy copytest to stdout (format json, force_not_null *); copy copytest to stdout (format json, force_null *); copy copytest to stdout (format json, on_error ignore); copy copytest from stdin(format json); -copy copytest (style) to stdout (format json); -- all of the above should yield error -- should fail: force_array requires json format @@ -110,6 +109,9 @@ copy copytest to stdout (format json, force_array); copy copytest to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- column list with json format +copy copytest (style, filler) to stdout (format json); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.43.0 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-06 09:38 jian he <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2026-03-06 09:38 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> COPY (SELECT 1 UNION ALL SELECT 2) TO stdout WITH (format json); still fails with v25-0002, json_tupledesc_ready is not helpful. I think I figured it out. We need to use BlessTupleDesc in BeginCopyTo. Then let slot->tts_tupleDescriptor point to cstate->queryDesc->tupDesc in CopyToJsonOneRow * CSV, text and json formats share the same TextLike routines except for the * one-row callback. This comment is not useful, I want to delete it. CopyToTextLikeStart + /* JSON-specific initialization */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + MemoryContext oldcxt; + + /* Allocate reusable JSON output buffer in long-lived context */ + oldcxt = MemoryContextSwitchTo(cstate->copycontext); + initStringInfo(&cstate->json_buf); + MemoryContextSwitchTo(oldcxt); + } We ca just add cstate->json_buf = makeStringInfo(); in BeginCopyTo. v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch added several fields to the CopyToStateData. Actually, there is a simpler way (construct a new Tupdesc and let composite_to_json do the job), please see my v26-0004. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v26-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (12.3K, 2-v26-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 61ea6a492ef900bf8745455847e2d9cffb396152 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Fri, 6 Mar 2026 14:47:58 +0800 Subject: [PATCH v26 3/4] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 41 ++++++++++++++++++++++++++++-- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 33 ++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 10 ++++++++ 7 files changed, 127 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29c121c7f08..84254d46a67 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index e87310ec5a0..4ea44daee0a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,7 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, it is * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ @@ -135,6 +136,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -176,7 +178,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -242,6 +244,18 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } + } } /* @@ -318,13 +332,24 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) @@ -345,6 +370,18 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) rowdata = ExecFetchSlotHeapTupleDatum(slot); composite_to_json(rowdata, cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); CopySendTextLikeEndOfRow(cstate); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 0d9649c1f0a..4b18cc6e2cd 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 4324e3e4961..309a33ca2e7 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -80,6 +80,16 @@ copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} +copy (select 1 union all select 2) to stdout with (format json, force_array true); +[ + {"?column?":1} +,{"?column?":2} +] +copy (values (1), (2)) TO stdout with (format json, force_array true); +[ + {"column1":1} +,{"column1":2} +] copy copytest to stdout json; {"style":"DOS","test":"abc\r\ndef","filler":1} {"style":"Unix","test":"abc\ndef","filler":2} @@ -122,6 +132,29 @@ ERROR: COPY JSON mode cannot be used with COPY FROM copy copytest (style) to stdout (format json); ERROR: column selection is not supported in JSON mode -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 4e9f74537f8..8a20907dd4c 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -85,6 +85,8 @@ copy copytest3 to stdout csv header; --- test copying in JSON mode with various styles copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); +copy (select 1 union all select 2) to stdout with (format json, force_array true); +copy (values (1), (2)) TO stdout with (format json, force_array true); copy copytest to stdout json; copy copytest to stdout (format json); @@ -105,6 +107,14 @@ copy copytest from stdin(format json); copy copytest (style) to stdout (format json); -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v26-0004-COPY-TO-JSON-support-column-lists.patch (6.1K, 3-v26-0004-COPY-TO-JSON-support-column-lists.patch) download | inline diff: From c531737641b7c0951b29b80144362168d1f96aa1 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Fri, 6 Mar 2026 17:30:23 +0800 Subject: [PATCH v26 4/4] COPY TO JSON support column lists Author: Andrew Dunstan <[email protected]> Reviewed-by: jian he <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copyto.c | 54 +++++++++++++++++++++++++----- src/test/regress/expected/copy.out | 15 +++++++-- src/test/regress/sql/copy.sql | 5 ++- 3 files changed, 63 insertions(+), 11 deletions(-) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 4ea44daee0a..a45ef6bcab3 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -89,6 +89,8 @@ typedef struct CopyToStateData bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, it is * initliazed in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for the COPY TO column list. + * This is only necessary for JSON output. */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -359,11 +361,17 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) /* * composite_to_json() requires a stable TupleDesc. The slot's descriptor * (slot->tts_tupleDescriptor) may change during the execution of a SELECT - * query, using cstate->queryDesc instead. No need worry this if COPY TO - * is directly from a table. + * query, using cstate->queryDesc instead. + * + * No need worry this if COPY TO is directly from a table, howeever when a + * direct COPY TO from a table with a subset of columns in JSON mode, the + * default slot's descriptor is obviously not OK, use the dedicated + * TupleDesc constructed in BeginCopyTO. */ - if (!cstate->rel) - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + if (cstate->rel && (RelationGetDescr(cstate->rel) != cstate->tupDesc)) + ReleaseTupleDesc(slot->tts_tupleDescriptor); + + slot->tts_tupleDescriptor = cstate->tupDesc; resetStringInfo(cstate->json_buf); @@ -839,6 +847,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = RelationGetDescr(cstate->rel); cstate->partitions = children; + cstate->tupDesc = tupDesc; } else { @@ -976,6 +985,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = cstate->queryDesc->tupDesc; tupDesc = BlessTupleDesc(tupDesc); + cstate->tupDesc = tupDesc; } /* Generate or convert list of attributes to process */ @@ -986,10 +996,38 @@ BeginCopyTo(ParseState *pstate, { cstate->json_buf = makeStringInfo(); - if (attnamelist != NIL) - ereport(ERROR, - errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column selection is not supported in JSON mode")); + if (attnamelist != NIL && rel) + { + char *attname; + Oid atttypid; + int32 atttypmod; + int attdim; + TupleDesc resultDesc; + + /* + * allocate a new tuple descriptor + */ + resultDesc = CreateTemplateTupleDesc(list_length(cstate->attnumlist)); + + foreach_int(attnum, cstate->attnumlist) + { + Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); + + attname = NameStr(attr->attname); + atttypid = attr->atttypid; + atttypmod = attr->atttypmod; + attdim = attr->attndims; + + TupleDescInitEntry(resultDesc, + foreach_current_index(attnum) + 1, + attname, + atttypid, + atttypmod, + attdim); + } + + cstate->tupDesc = BlessTupleDesc(resultDesc); + } } num_phys_attrs = tupDesc->natts; diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 309a33ca2e7..3ffb08bfada 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -129,8 +129,6 @@ copy copytest to stdout (format json, reject_limit 1); ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE copy copytest from stdin(format json); ERROR: COPY JSON mode cannot be used with COPY FROM -copy copytest (style) to stdout (format json); -ERROR: column selection is not supported in JSON mode -- all of the above should yield error -- should fail: force_array requires json format copy copytest to stdout (format csv, force_array true); @@ -155,6 +153,19 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with json format +copy copytest (style, filler) to stdout (format json); +{"style":"DOS","filler":1667391763} +{"style":"Unix","filler":1701055075} +{"style":"Mac","filler":1667391761} +{"style":"esc\\ape","filler":1918656791} +copy copytest (style, filler) to stdout (format json, force_array true); +[ + {"style":"DOS","filler":1667391763} +,{"style":"Unix","filler":1701055075} +,{"style":"Mac","filler":1667391761} +,{"style":"esc\\ape","filler":1918656791} +] -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 8a20907dd4c..dae97bebb7d 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -104,7 +104,6 @@ copy copytest to stdout (format json, force_null *); copy copytest to stdout (format json, on_error ignore); copy copytest to stdout (format json, reject_limit 1); copy copytest from stdin(format json); -copy copytest (style) to stdout (format json); -- all of the above should yield error -- should fail: force_array requires json format @@ -115,6 +114,10 @@ copy copytest to stdout (format json, force_array); copy copytest to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- column list with json format +copy copytest (style, filler) to stdout (format json); +copy copytest (style, filler) to stdout (format json, force_array true); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v26-0002-json-format-for-COPY-TO.patch (23.3K, 4-v26-0002-json-format-for-COPY-TO.patch) download | inline diff: From 7244131f4710228f3e3998524dcd930e349e00bd Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Fri, 6 Mar 2026 14:32:48 +0800 Subject: [PATCH v26 2/4] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 49 ++++++++++++----- src/backend/commands/copyto.c | 85 +++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 +++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 86 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 51 ++++++++++++++++++ 10 files changed, 271 insertions(+), 31 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29c121c7f08 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 0325a16f82a..e87310ec5a0 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -84,6 +86,8 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + StringInfo json_buf; /* reusable buffer for JSON output, it is + * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -130,6 +134,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -148,9 +153,6 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. - * - * CSV and text formats share the same TextLike routines except for the - * one-row callback. */ /* text format */ @@ -169,6 +171,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +195,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +221,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +245,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +318,38 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + /* + * composite_to_json() requires a stable TupleDesc. The slot's descriptor + * (slot->tts_tupleDescriptor) may change during the execution of a SELECT + * query, using cstate->queryDesc instead. No need worry this if COPY TO + * is directly from a table. + */ + if (!cstate->rel) + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + + resetStringInfo(cstate->json_buf); + + rowdata = ExecFetchSlotHeapTupleDatum(slot); + composite_to_json(rowdata, cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +441,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -506,7 +559,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -885,11 +938,23 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; + tupDesc = BlessTupleDesc(tupDesc); } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* JSON outputs whole rows; a column list doesn't make sense */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + cstate->json_buf = makeStringInfo(); + + if (attnamelist != NIL) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column selection is not supported in JSON mode")); + } + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 3c3e24324a8..40ad9073901 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3612,6 +3612,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3694,6 +3698,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index f8c0865ca89..0d9649c1f0a 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..4324e3e4961 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,92 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +{"?column?":1} +{"?column?":2} +copy (values (1), (2)) TO stdout with (format json); +{"column1":1} +{"column1":2} +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest to stdout (format json, reject_limit 1); +ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE +copy copytest from stdin(format json); +ERROR: COPY JSON mode cannot be used with COPY FROM +copy copytest (style) to stdout (format json); +ERROR: column selection is not supported in JSON mode +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..4e9f74537f8 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,57 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +copy (values (1), (2)) TO stdout with (format json); +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest to stdout (format json, reject_limit 1); +copy copytest from stdin(format json); +copy copytest (style) to stdout (format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 [text/x-patch] v26-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 5-v26-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From e4747ff8a87e79edb8fd3b7778bcbe8a3a6e85f7 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v26 1/4] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2f42f55e229..4d927410159 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index fbd13353efc..c366874bd95 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -172,7 +172,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -750,7 +750,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -777,7 +777,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9ceeff6d99e..0325a16f82a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 77e3c04144e..8399be97fd5 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-06 12:33 Andrew Dunstan <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Andrew Dunstan @ 2026-03-06 12:33 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-03-06 Fr 4:38 AM, jian he wrote: > COPY (SELECT 1 UNION ALL SELECT 2) TO stdout WITH (format json); > > still fails with v25-0002, json_tupledesc_ready is not helpful. > I think I figured it out. We need to use BlessTupleDesc in BeginCopyTo. > Then let slot->tts_tupleDescriptor point to cstate->queryDesc->tupDesc > in CopyToJsonOneRow > > * CSV, text and json formats share the same TextLike routines except for the > * one-row callback. > This comment is not useful, I want to delete it. > > CopyToTextLikeStart > + /* JSON-specific initialization */ > + if (cstate->opts.format == COPY_FORMAT_JSON) > + { > + MemoryContext oldcxt; > + > + /* Allocate reusable JSON output buffer in long-lived context */ > + oldcxt = MemoryContextSwitchTo(cstate->copycontext); > + initStringInfo(&cstate->json_buf); > + MemoryContextSwitchTo(oldcxt); > + } > We ca just add > > cstate->json_buf = makeStringInfo(); > > in BeginCopyTo. > > v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch > added several fields to the CopyToStateData. > Actually, there is a simpler way (construct a new Tupdesc and let > composite_to_json do the job), please see my v26-0004. > Yeah, I got some crashes using this, but reworked it a bit, and all now seems good. I was apparently mistaken about the supposed speedup, but the performance is steady. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com Attachments: [text/x-patch] v27-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 2-v27-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From e94cc2d8543153d4ab671b33e4494d4428ef005c Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v27 1/4] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2f42f55e229..4d927410159 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index fbd13353efc..c366874bd95 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -172,7 +172,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -750,7 +750,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -777,7 +777,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9ceeff6d99e..0325a16f82a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -181,9 +181,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -220,7 +220,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -397,7 +397,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 77e3c04144e..8399be97fd5 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.43.0 [text/x-patch] v27-0002-json-format-for-COPY-TO.patch (23.3K, 3-v27-0002-json-format-for-COPY-TO.patch) download | inline diff: From b3222f265bd157690d5f69ea8eba84be13e5910a Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Fri, 6 Mar 2026 14:32:48 +0800 Subject: [PATCH v27 2/4] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 49 ++++++++++++----- src/backend/commands/copyto.c | 85 +++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 +++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 86 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 51 ++++++++++++++++++ 10 files changed, 271 insertions(+), 31 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29c121c7f08 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 0325a16f82a..e87310ec5a0 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -84,6 +86,8 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + StringInfo json_buf; /* reusable buffer for JSON output, it is + * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -130,6 +134,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -148,9 +153,6 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. - * - * CSV and text formats share the same TextLike routines except for the - * one-row callback. */ /* text format */ @@ -169,6 +171,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -185,12 +195,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -209,6 +221,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -231,7 +245,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -304,13 +318,38 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + /* + * composite_to_json() requires a stable TupleDesc. The slot's descriptor + * (slot->tts_tupleDescriptor) may change during the execution of a SELECT + * query, using cstate->queryDesc instead. No need worry this if COPY TO + * is directly from a table. + */ + if (!cstate->rel) + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + + resetStringInfo(cstate->json_buf); + + rowdata = ExecFetchSlotHeapTupleDatum(slot); + composite_to_json(rowdata, cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -402,9 +441,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -506,7 +559,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -885,11 +938,23 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; + tupDesc = BlessTupleDesc(tupDesc); } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* JSON outputs whole rows; a column list doesn't make sense */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + cstate->json_buf = makeStringInfo(); + + if (attnamelist != NIL) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column selection is not supported in JSON mode")); + } + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 3c3e24324a8..40ad9073901 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3612,6 +3612,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3694,6 +3698,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index f8c0865ca89..0d9649c1f0a 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..4324e3e4961 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,92 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +{"?column?":1} +{"?column?":2} +copy (values (1), (2)) TO stdout with (format json); +{"column1":1} +{"column1":2} +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest to stdout (format json, reject_limit 1); +ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE +copy copytest from stdin(format json); +ERROR: COPY JSON mode cannot be used with COPY FROM +copy copytest (style) to stdout (format json); +ERROR: column selection is not supported in JSON mode +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..4e9f74537f8 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,57 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +copy (values (1), (2)) TO stdout with (format json); +copy copytest to stdout json; +copy copytest to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest to stdout (format json, reject_limit 1); +copy copytest from stdin(format json); +copy copytest (style) to stdout (format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.43.0 [text/x-patch] v27-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (12.3K, 4-v27-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 6c338f2f14f8d00be0204752e066373ab86d96d3 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Fri, 6 Mar 2026 14:47:58 +0800 Subject: [PATCH v27 3/4] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 41 ++++++++++++++++++++++++++++-- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 33 ++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 10 ++++++++ 7 files changed, 127 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29c121c7f08..84254d46a67 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index e87310ec5a0..4ea44daee0a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -86,6 +86,7 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, it is * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ @@ -135,6 +136,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -176,7 +178,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -242,6 +244,18 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } + } } /* @@ -318,13 +332,24 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) @@ -345,6 +370,18 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) rowdata = ExecFetchSlotHeapTupleDatum(slot); composite_to_json(rowdata, cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); CopySendTextLikeEndOfRow(cstate); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 0d9649c1f0a..4b18cc6e2cd 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 4324e3e4961..309a33ca2e7 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -80,6 +80,16 @@ copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} +copy (select 1 union all select 2) to stdout with (format json, force_array true); +[ + {"?column?":1} +,{"?column?":2} +] +copy (values (1), (2)) TO stdout with (format json, force_array true); +[ + {"column1":1} +,{"column1":2} +] copy copytest to stdout json; {"style":"DOS","test":"abc\r\ndef","filler":1} {"style":"Unix","test":"abc\ndef","filler":2} @@ -122,6 +132,29 @@ ERROR: COPY JSON mode cannot be used with COPY FROM copy copytest (style) to stdout (format json); ERROR: column selection is not supported in JSON mode -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 4e9f74537f8..8a20907dd4c 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -85,6 +85,8 @@ copy copytest3 to stdout csv header; --- test copying in JSON mode with various styles copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); +copy (select 1 union all select 2) to stdout with (format json, force_array true); +copy (values (1), (2)) TO stdout with (format json, force_array true); copy copytest to stdout json; copy copytest to stdout (format json); @@ -105,6 +107,14 @@ copy copytest from stdin(format json); copy copytest (style) to stdout (format json); -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.43.0 [text/x-patch] v27-0004-COPY-TO-JSON-support-column-lists.patch (12.5K, 5-v27-0004-COPY-TO-JSON-support-column-lists.patch) download | inline diff: From a627df0760d9803dea491798efe1298bff6aefcb Mon Sep 17 00:00:00 2001 From: Andrew Dunstan <[email protected]> Date: Fri, 6 Mar 2026 06:06:28 -0500 Subject: [PATCH v27 4/4] COPY TO JSON: support column lists When a column list is specified (e.g. COPY t (a, b) TO ... FORMAT json), build a projected TupleDesc containing only the selected columns and form a new tuple per row via heap_form_tuple(), so that composite_to_json() emits the correct column names and values. Use HeapTupleHeaderGetDatum() directly on the formed tuple rather than heap_copy_tuple_as_datum(), since heap_form_tuple() already stamps the datum-length, type-id, and type-mod fields on t_data, avoiding an unnecessary palloc+memcpy per row. Add regression tests covering column lists with diverse data types including json, jsonb, int[], numeric, boolean, timestamp, and text, exercising various column subsets and NULL handling. Author: Andrew Dunstan <[email protected]> Reviewed-by: jian he <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copyto.c | 109 ++++++++++++++++++++++++----- src/test/regress/expected/copy.out | 66 ++++++++++++++++- src/test/regress/sql/copy.sql | 39 ++++++++++- 3 files changed, 193 insertions(+), 21 deletions(-) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 4ea44daee0a..992f3970d1a 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -87,8 +87,12 @@ typedef struct CopyToStateData char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ bool json_row_delim_needed; /* need delimiter before next row */ - StringInfo json_buf; /* reusable buffer for JSON output, it is - * initliazed in BeginCopyTo */ + StringInfo json_buf; /* reusable buffer for JSON output, + * initialized in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for JSON output; for a column + * list this is a projected descriptor */ + Datum *json_projvalues; /* pre-allocated projection values, or NULL */ + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -356,18 +360,51 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) { Datum rowdata; - /* - * composite_to_json() requires a stable TupleDesc. The slot's descriptor - * (slot->tts_tupleDescriptor) may change during the execution of a SELECT - * query, using cstate->queryDesc instead. No need worry this if COPY TO - * is directly from a table. - */ - if (!cstate->rel) - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; - resetStringInfo(cstate->json_buf); - rowdata = ExecFetchSlotHeapTupleDatum(slot); + if (cstate->json_projvalues != NULL) + { + /* + * Column list case: project selected column values into sequential + * positions matching the custom TupleDesc, then form a new tuple. + */ + HeapTuple tup; + int i = 0; + + foreach_int(attnum, cstate->attnumlist) + { + cstate->json_projvalues[i] = slot->tts_values[attnum - 1]; + cstate->json_projnulls[i] = slot->tts_isnull[attnum - 1]; + i++; + } + + tup = heap_form_tuple(cstate->tupDesc, + cstate->json_projvalues, + cstate->json_projnulls); + + /* + * heap_form_tuple already stamps the datum-length, type-id, and + * type-mod fields on t_data, so we can use it directly as a + * composite Datum without the extra palloc+memcpy that + * heap_copy_tuple_as_datum would do. Any TOAST pointers in the + * projected values will be detoasted by the per-column output + * functions called from composite_to_json. + */ + rowdata = HeapTupleHeaderGetDatum(tup->t_data); + } + else + { + /* + * Full table or query without column list. Ensure the slot uses + * cstate->tupDesc so that the datum is stamped with the right type; + * for queries this must be the blessed descriptor so that + * composite_to_json can look it up via lookup_rowtype_tupdesc. + * For relations the assignment is a no-op (same pointer). + */ + slot->tts_tupleDescriptor = cstate->tupDesc; + rowdata = ExecFetchSlotHeapTupleDatum(slot); + } + composite_to_json(rowdata, cstate->json_buf, false); if (cstate->opts.force_array) @@ -839,6 +876,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = RelationGetDescr(cstate->rel); cstate->partitions = children; + cstate->tupDesc = tupDesc; } else { @@ -975,21 +1013,56 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; - tupDesc = BlessTupleDesc(tupDesc); + + /* + * For JSON format, bless the TupleDesc so that composite_to_json() + * can look up the RECORD type via lookup_rowtype_tupdesc(). + */ + if (cstate->opts.format == COPY_FORMAT_JSON) + tupDesc = BlessTupleDesc(tupDesc); + cstate->tupDesc = tupDesc; } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); - /* JSON outputs whole rows; a column list doesn't make sense */ + /* Set up JSON-specific state */ if (cstate->opts.format == COPY_FORMAT_JSON) { cstate->json_buf = makeStringInfo(); - if (attnamelist != NIL) - ereport(ERROR, - errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column selection is not supported in JSON mode")); + if (attnamelist != NIL && rel) + { + int natts = list_length(cstate->attnumlist); + TupleDesc resultDesc; + + /* + * Build a TupleDesc describing only the selected columns so that + * composite_to_json() emits the right column names and types. + */ + resultDesc = CreateTemplateTupleDesc(natts); + + foreach_int(attnum, cstate->attnumlist) + { + Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); + + TupleDescInitEntry(resultDesc, + foreach_current_index(attnum) + 1, + NameStr(attr->attname), + attr->atttypid, + attr->atttypmod, + attr->attndims); + } + + cstate->tupDesc = BlessTupleDesc(resultDesc); + + /* + * Pre-allocate arrays for projecting selected column values into + * sequential positions matching the custom TupleDesc. + */ + cstate->json_projvalues = (Datum *) palloc(natts * sizeof(Datum)); + cstate->json_projnulls = (bool *) palloc(natts * sizeof(bool)); + } } num_phys_attrs = tupDesc->natts; diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 309a33ca2e7..645bdc1a21c 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -77,6 +77,9 @@ c1,"col with , comma","col with "" quote" copy (select 1 union all select 2) to stdout with (format json); {"?column?":1} {"?column?":2} +copy (select 1 as foo union all select 2) to stdout with (format json); +{"foo":1} +{"foo":2} copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} @@ -129,8 +132,6 @@ copy copytest to stdout (format json, reject_limit 1); ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE copy copytest from stdin(format json); ERROR: COPY JSON mode cannot be used with COPY FROM -copy copytest (style) to stdout (format json); -ERROR: column selection is not supported in JSON mode -- all of the above should yield error -- should fail: force_array requires json format copy copytest to stdout (format csv, force_array true); @@ -155,6 +156,67 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with json format +copy copytest (style, filler) to stdout (format json); +{"style":"DOS","filler":1} +{"style":"Unix","filler":2} +{"style":"Mac","filler":3} +{"style":"esc\\ape","filler":4} +copy copytest (style, filler) to stdout (format json, force_array true); +[ + {"style":"DOS","filler":1} +,{"style":"Unix","filler":2} +,{"style":"Mac","filler":3} +,{"style":"esc\\ape","filler":4} +] +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); +-- full table +copy copyjsontest_types to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2},"arr":[1,2,3],"n":3.14,"b":true,"ts":"2024-01-15T10:30:00","t":"hello"} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}},"arr":[4,5],"n":-99.99,"b":false,"ts":"2024-06-30T23:59:59","t":"world"} +{"id":3,"js":null,"jsb":null,"arr":[],"n":null,"b":null,"ts":null,"t":null} +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2}} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}}} +{"id":3,"js":null,"jsb":null} +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +{"id":1,"arr":[1,2,3],"n":3.14,"b":true} +{"id":2,"arr":[4,5],"n":-99.99,"b":false} +{"id":3,"arr":[],"n":null,"b":null} +copy copyjsontest_types (jsb, t) to stdout (format json); +{"jsb":{"b": 2},"t":"hello"} +{"jsb":{"nested": {"k": "v"}},"t":"world"} +{"jsb":null,"t":null} +copy copyjsontest_types (id, ts) to stdout (format json); +{"id":1,"ts":"2024-01-15T10:30:00"} +{"id":2,"ts":"2024-06-30T23:59:59"} +{"id":3,"ts":null} +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +{"js":{"a":1}} +{"js":[1,null,"x"]} +{"js":null} +copy copyjsontest_types (jsb) to stdout (format json); +{"jsb":{"b": 2}} +{"jsb":{"nested": {"k": "v"}}} +{"jsb":null} +drop table copyjsontest_types; -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 8a20907dd4c..c4d83762805 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -84,6 +84,7 @@ copy copytest3 to stdout csv header; --- test copying in JSON mode with various styles copy (select 1 union all select 2) to stdout with (format json); +copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); copy (select 1 union all select 2) to stdout with (format json, force_array true); copy (values (1), (2)) TO stdout with (format json, force_array true); @@ -104,7 +105,6 @@ copy copytest to stdout (format json, force_null *); copy copytest to stdout (format json, on_error ignore); copy copytest to stdout (format json, reject_limit 1); copy copytest from stdin(format json); -copy copytest (style) to stdout (format json); -- all of the above should yield error -- should fail: force_array requires json format @@ -115,6 +115,43 @@ copy copytest to stdout (format json, force_array); copy copytest to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- column list with json format +copy copytest (style, filler) to stdout (format json); +copy copytest (style, filler) to stdout (format json, force_array true); + +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); + +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); + +-- full table +copy copyjsontest_types to stdout (format json); + +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +copy copyjsontest_types (jsb, t) to stdout (format json); +copy copyjsontest_types (id, ts) to stdout (format json); + +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +copy copyjsontest_types (jsb) to stdout (format json); + +drop table copyjsontest_types; + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.43.0 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-08 16:16 jian he <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: jian he @ 2026-03-08 16:16 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> hi. V27-0002 is still not bullet-proof. drop table if exists t1; create table t1(a int); insert into t1 values (1); copy (select * from t1) to stdout json; {"a":1} WARNING: resource was not closed: TupleDesc 0x7171d0ca3440 (18239,-1) Also see ExecAssignScanProjectionInfo->ExecConditionalAssignProjectionInfo So in v28-0002, I changed to + /* + * composite_to_json() requires a stable TupleDesc. Since the slot's + * descriptor (slot->tts_tupleDescriptor) can change during the execution + * of a SELECT query, we use cstate->queryDesc->tupDesc instead. This + * precaution is only necessary when the output slot's TupleDesc is of + * type RECORDOID. + */ + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + cstate->json_projvalues = (Datum *) palloc(natts * sizeof(Datum)); + cstate->json_projnulls = (bool *) palloc(natts * sizeof(bool)); I changed it to + cstate->json_projvalues = palloc_array(Datum, natts); + cstate->json_projnulls = palloc_array(bool, natts); + rowdata = HeapTupleHeaderGetDatum(tup->t_data); I changed it to + rowdata = HeapTupleGetDatum(tup); Patch v28-0004 adds the json_projvalues and json_projnulls pointers to struct CopyToStateData. I wondered if adding these would slow the COPY TO with TEXT and CSV format, so I ran a quick test using a 36-column table. Surprisingly, v28 actually make COPY TO with TEXT and CSV performs a little bit faster. But I didn't find out why. You may also try the attached test script: copyto_json_perfomance_test.nocfbot. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v28-0002-json-format-for-COPY-TO.patch (23.7K, 2-v28-0002-json-format-for-COPY-TO.patch) download | inline diff: From 439fc107c50408f51e3192f221373e1333672063 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Sun, 8 Mar 2026 23:44:11 +0800 Subject: [PATCH v28 2/4] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Andrew Dunstan <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 +++-- src/backend/commands/copy.c | 49 +++++++++++----- src/backend/commands/copyto.c | 86 ++++++++++++++++++++++++---- src/backend/parser/gram.y | 8 +++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 91 ++++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 52 +++++++++++++++++ 10 files changed, 278 insertions(+), 31 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29c121c7f08 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index b0ee91fc9c1..6971f4b85af 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -85,6 +87,8 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + StringInfo json_buf; /* reusable buffer for JSON output, it is + * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -131,6 +135,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,9 +154,6 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. - * - * CSV and text formats share the same TextLike routines except for the - * one-row callback. */ /* text format */ @@ -170,6 +172,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -186,12 +196,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -210,6 +222,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -232,7 +246,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -305,13 +319,39 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + /* + * composite_to_json() requires a stable TupleDesc. Since the slot's + * descriptor (slot->tts_tupleDescriptor) can change during the execution + * of a SELECT query, we use cstate->queryDesc->tupDesc instead. This + * precaution is only necessary when the output slot's TupleDesc is of + * type RECORDOID. + */ + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + + resetStringInfo(cstate->json_buf); + + rowdata = ExecFetchSlotHeapTupleDatum(slot); + composite_to_json(rowdata, cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -403,9 +443,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -507,7 +561,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -886,11 +940,23 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; + tupDesc = BlessTupleDesc(tupDesc); } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* JSON outputs whole rows; a column list doesn't make sense */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + cstate->json_buf = makeStringInfo(); + + if (attnamelist != NIL) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column selection is not supported in JSON mode")); + } + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 9cbe8eafc45..136fd19b854 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3612,6 +3612,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3694,6 +3698,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 6484c6a3dd4..bb82bdbcc48 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..72011f3492c 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,97 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +{"?column?":1} +{"?column?":2} +copy (values (1), (2)) TO stdout with (format json); +{"column1":1} +{"column1":2} +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy (select * from copytest) to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest to stdout (format json, reject_limit 1); +ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE +copy copytest from stdin(format json); +ERROR: COPY JSON mode cannot be used with COPY FROM +copy copytest (style) to stdout (format json); +ERROR: column selection is not supported in JSON mode +-- all of the above should yield error +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..a6c923ef5ab 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,58 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +copy (values (1), (2)) TO stdout with (format json); +copy copytest to stdout json; +copy copytest to stdout (format json); +copy (select * from copytest) to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest to stdout (format json, reject_limit 1); +copy copytest from stdin(format json); +copy copytest (style) to stdout (format json); +-- all of the above should yield error + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 [text/x-patch] v28-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 3-v28-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From 226c834e93351ffda039774fbd4b57a9ab16b4f7 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v28 1/4] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2f42f55e229..4d927410159 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 84c8809a889..5f0c551e7ec 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -173,7 +173,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -751,7 +751,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -778,7 +778,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index d6ef7275a64..b0ee91fc9c1 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -182,9 +182,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -221,7 +221,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -398,7 +398,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3250564d4ff..520cdd36800 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 [text/x-patch] v28-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (12.3K, 4-v28-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 247eceeeaec62677d899f225bd83905da21f0087 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Sun, 8 Mar 2026 15:54:23 +0800 Subject: [PATCH v28 3/4] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 41 ++++++++++++++++++++++++++++-- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 33 ++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 10 ++++++++ 7 files changed, 127 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29c121c7f08..84254d46a67 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 6971f4b85af..38fbf7d4424 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -87,6 +87,7 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, it is * initliazed in BeginCopyTo */ copy_data_dest_cb data_dest_cb; /* function for writing data */ @@ -136,6 +137,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -177,7 +179,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -243,6 +245,18 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } + } } /* @@ -319,13 +333,24 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) @@ -347,6 +372,18 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) rowdata = ExecFetchSlotHeapTupleDatum(slot); composite_to_json(rowdata, cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); CopySendTextLikeEndOfRow(cstate); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index bb82bdbcc48..00458cfb4bc 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 72011f3492c..e1d51335e33 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -80,6 +80,16 @@ copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} +copy (select 1 union all select 2) to stdout with (format json, force_array true); +[ + {"?column?":1} +,{"?column?":2} +] +copy (values (1), (2)) TO stdout with (format json, force_array true); +[ + {"column1":1} +,{"column1":2} +] copy copytest to stdout json; {"style":"DOS","test":"abc\r\ndef","filler":1} {"style":"Unix","test":"abc\ndef","filler":2} @@ -127,6 +137,29 @@ ERROR: COPY JSON mode cannot be used with COPY FROM copy copytest (style) to stdout (format json); ERROR: column selection is not supported in JSON mode -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index a6c923ef5ab..764d19f4947 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -85,6 +85,8 @@ copy copytest3 to stdout csv header; --- test copying in JSON mode with various styles copy (select 1 union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); +copy (select 1 union all select 2) to stdout with (format json, force_array true); +copy (values (1), (2)) TO stdout with (format json, force_array true); copy copytest to stdout json; copy copytest to stdout (format json); copy (select * from copytest) to stdout (format json); @@ -106,6 +108,14 @@ copy copytest from stdin(format json); copy copytest (style) to stdout (format json); -- all of the above should yield error +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [text/x-patch] v28-0004-COPY-TO-JSON-support-column-lists.patch (12.7K, 5-v28-0004-COPY-TO-JSON-support-column-lists.patch) download | inline diff: From 365f340a0b733a3d9b5fdf540a2623c3ea9d4d8d Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Sun, 8 Mar 2026 23:58:29 +0800 Subject: [PATCH v28 4/4] COPY TO JSON: support column lists When a column list is specified (e.g. COPY t (a, b) TO ... FORMAT json), build a projected TupleDesc containing only the selected columns and form a new tuple per row via heap_form_tuple(), so that composite_to_json() emits the correct column names and values. Use HeapTupleHeaderGetDatum() directly on the formed tuple rather than heap_copy_tuple_as_datum(), since heap_form_tuple() already stamps the datum-length, type-id, and type-mod fields on t_data, avoiding an unnecessary palloc+memcpy per row. Add regression tests covering column lists with diverse data types including json, jsonb, int[], numeric, boolean, timestamp, and text, exercising various column subsets and NULL handling. Author: Andrew Dunstan <[email protected]> Reviewed-by: jian he <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copyto.c | 105 ++++++++++++++++++++++++----- src/test/regress/expected/copy.out | 73 +++++++++++++++++++- src/test/regress/sql/copy.sql | 40 ++++++++++- 3 files changed, 197 insertions(+), 21 deletions(-) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 38fbf7d4424..faa8e323f56 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -88,8 +88,13 @@ typedef struct CopyToStateData char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ bool json_row_delim_needed; /* need delimiter before next row */ - StringInfo json_buf; /* reusable buffer for JSON output, it is - * initliazed in BeginCopyTo */ + StringInfo json_buf; /* reusable buffer for JSON output, + * initialized in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for JSON output; for a column + * list this is a projected descriptor */ + Datum *json_projvalues; /* pre-allocated projection values, or + * NULL */ + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -357,19 +362,53 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) { Datum rowdata; - /* - * composite_to_json() requires a stable TupleDesc. Since the slot's - * descriptor (slot->tts_tupleDescriptor) can change during the execution - * of a SELECT query, we use cstate->queryDesc->tupDesc instead. This - * precaution is only necessary when the output slot's TupleDesc is of - * type RECORDOID. - */ - if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; - resetStringInfo(cstate->json_buf); - rowdata = ExecFetchSlotHeapTupleDatum(slot); + if (cstate->json_projvalues != NULL) + { + /* + * Column list case: project selected column values into sequential + * positions matching the custom TupleDesc, then form a new tuple. + */ + HeapTuple tup; + int i = 0; + + foreach_int(attnum, cstate->attnumlist) + { + cstate->json_projvalues[i] = slot->tts_values[attnum - 1]; + cstate->json_projnulls[i] = slot->tts_isnull[attnum - 1]; + i++; + } + + tup = heap_form_tuple(cstate->tupDesc, + cstate->json_projvalues, + cstate->json_projnulls); + + /* + * heap_form_tuple already stamps the datum-length, type-id, and + * type-mod fields on t_data, so we can use it directly as a composite + * Datum without the extra pallocmemcpy that heap_copy_tuple_as_datum + * would do. Any TOAST pointers in the projected values will be + * detoasted by the per-column output functions called from + * composite_to_json. + */ + rowdata = HeapTupleGetDatum(tup); + } + else + { + /* + * Full table or query without column list. Ensure the slot uses + * cstate->tupDesc so that the datum is stamped with the right type; + * for queries output type is RECORDOID this must be the blessed + * descriptor so that composite_to_json can look it up via + * lookup_rowtype_tupdesc. + */ + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + + rowdata = ExecFetchSlotHeapTupleDatum(slot); + } + composite_to_json(rowdata, cstate->json_buf, false); if (cstate->opts.force_array) @@ -841,6 +880,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = RelationGetDescr(cstate->rel); cstate->partitions = children; + cstate->tupDesc = tupDesc; } else { @@ -978,20 +1018,49 @@ BeginCopyTo(ParseState *pstate, tupDesc = cstate->queryDesc->tupDesc; tupDesc = BlessTupleDesc(tupDesc); + cstate->tupDesc = tupDesc; } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); - /* JSON outputs whole rows; a column list doesn't make sense */ + /* Set up JSON-specific state */ if (cstate->opts.format == COPY_FORMAT_JSON) { cstate->json_buf = makeStringInfo(); - if (attnamelist != NIL) - ereport(ERROR, - errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("column selection is not supported in JSON mode")); + if (attnamelist != NIL && rel) + { + int natts = list_length(cstate->attnumlist); + TupleDesc resultDesc; + + /* + * Build a TupleDesc describing only the selected columns so that + * composite_to_json() emits the right column names and types. + */ + resultDesc = CreateTemplateTupleDesc(natts); + + foreach_int(attnum, cstate->attnumlist) + { + Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); + + TupleDescInitEntry(resultDesc, + foreach_current_index(attnum) + 1, + NameStr(attr->attname), + attr->atttypid, + attr->atttypmod, + attr->attndims); + } + + cstate->tupDesc = BlessTupleDesc(resultDesc); + + /* + * Pre-allocate arrays for projecting selected column values into + * sequential positions matching the custom TupleDesc. + */ + cstate->json_projvalues = palloc_array(Datum, natts); + cstate->json_projnulls = palloc_array(bool, natts); + } } num_phys_attrs = tupDesc->natts; diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index e1d51335e33..e44b4a1d79d 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -77,6 +77,9 @@ c1,"col with , comma","col with "" quote" copy (select 1 union all select 2) to stdout with (format json); {"?column?":1} {"?column?":2} +copy (select 1 as foo union all select 2) to stdout with (format json); +{"foo":1} +{"foo":2} copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} @@ -134,8 +137,6 @@ copy copytest to stdout (format json, reject_limit 1); ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE copy copytest from stdin(format json); ERROR: COPY JSON mode cannot be used with COPY FROM -copy copytest (style) to stdout (format json); -ERROR: column selection is not supported in JSON mode -- all of the above should yield error -- should fail: force_array requires json format copy copytest to stdout (format csv, force_array true); @@ -160,6 +161,74 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with json format +copy copytest (style, filler) to stdout (format json); +{"style":"DOS","filler":1} +{"style":"Unix","filler":2} +{"style":"Mac","filler":3} +{"style":"esc\\ape","filler":4} +copy copytest (style, filler) to stdout (format json, force_array true); +[ + {"style":"DOS","filler":1} +,{"style":"Unix","filler":2} +,{"style":"Mac","filler":3} +,{"style":"esc\\ape","filler":4} +] +copy copytest (style, test, filler) to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); +-- full table +copy copyjsontest_types to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2},"arr":[1,2,3],"n":3.14,"b":true,"ts":"2024-01-15T10:30:00","t":"hello"} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}},"arr":[4,5],"n":-99.99,"b":false,"ts":"2024-06-30T23:59:59","t":"world"} +{"id":3,"js":null,"jsb":null,"arr":[],"n":null,"b":null,"ts":null,"t":null} +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2}} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}}} +{"id":3,"js":null,"jsb":null} +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +{"id":1,"arr":[1,2,3],"n":3.14,"b":true} +{"id":2,"arr":[4,5],"n":-99.99,"b":false} +{"id":3,"arr":[],"n":null,"b":null} +copy copyjsontest_types (jsb, t) to stdout (format json); +{"jsb":{"b": 2},"t":"hello"} +{"jsb":{"nested": {"k": "v"}},"t":"world"} +{"jsb":null,"t":null} +copy copyjsontest_types (id, ts) to stdout (format json); +{"id":1,"ts":"2024-01-15T10:30:00"} +{"id":2,"ts":"2024-06-30T23:59:59"} +{"id":3,"ts":null} +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +{"js":{"a":1}} +{"js":[1,null,"x"]} +{"js":null} +copy copyjsontest_types (jsb) to stdout (format json); +{"jsb":{"b": 2}} +{"jsb":{"nested": {"k": "v"}}} +{"jsb":null} +drop table copyjsontest_types; -- embedded escaped characters create temp table copyjsontest ( id bigserial, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 764d19f4947..e4e70a82ecc 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -84,6 +84,7 @@ copy copytest3 to stdout csv header; --- test copying in JSON mode with various styles copy (select 1 union all select 2) to stdout with (format json); +copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); copy (select 1 union all select 2) to stdout with (format json, force_array true); copy (values (1), (2)) TO stdout with (format json, force_array true); @@ -105,7 +106,6 @@ copy copytest to stdout (format json, force_null *); copy copytest to stdout (format json, on_error ignore); copy copytest to stdout (format json, reject_limit 1); copy copytest from stdin(format json); -copy copytest (style) to stdout (format json); -- all of the above should yield error -- should fail: force_array requires json format @@ -116,6 +116,44 @@ copy copytest to stdout (format json, force_array); copy copytest to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- column list with json format +copy copytest (style, filler) to stdout (format json); +copy copytest (style, filler) to stdout (format json, force_array true); +copy copytest (style, test, filler) to stdout (format json, force_array true); + +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); + +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); + +-- full table +copy copyjsontest_types to stdout (format json); + +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +copy copyjsontest_types (jsb, t) to stdout (format json); +copy copyjsontest_types (id, ts) to stdout (format json); + +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +copy copyjsontest_types (jsb) to stdout (format json); + +drop table copyjsontest_types; + -- embedded escaped characters create temp table copyjsontest ( id bigserial, -- 2.34.1 [application/octet-stream] copyto_json_perfomance_test.nocfbot (1.7K, 6-copyto_json_perfomance_test.nocfbot) download ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-08 19:44 Andrew Dunstan <[email protected]> parent: jian he <[email protected]> 0 siblings, 1 reply; 22+ messages in thread From: Andrew Dunstan @ 2026-03-08 19:44 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-03-08 Su 12:16 PM, jian he wrote: > hi. > > V27-0002 is still not bullet-proof. > > drop table if exists t1; > create table t1(a int); > insert into t1 values (1); > copy (select * from t1) to stdout json; > {"a":1} > WARNING: resource was not closed: TupleDesc 0x7171d0ca3440 (18239,-1) > > Also see ExecAssignScanProjectionInfo->ExecConditionalAssignProjectionInfo > So in v28-0002, I changed to > + /* > + * composite_to_json() requires a stable TupleDesc. Since the slot's > + * descriptor (slot->tts_tupleDescriptor) can change during the execution > + * of a SELECT query, we use cstate->queryDesc->tupDesc instead. This > + * precaution is only necessary when the output slot's TupleDesc is of > + * type RECORDOID. > + */ > + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) > + slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? How about something like this?: - * Full table or query without column list. Ensure the slot uses - * cstate->tupDesc so that the datum is stamped with the right type; - * for queries output type is RECORDOID this must be the blessed - * descriptor so that composite_to_json can look it up via - * lookup_rowtype_tupdesc. + * Full table or query without column list. For queries, the slot's + * TupleDesc may carry RECORDOID, which is not registered in the type + * cache and would cause composite_to_json's lookup_rowtype_tupdesc + * call to fail. Build a HeapTuple stamped with the blessed + * descriptor so the type can be looked up correctly. */ if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; + { + HeapTuple tup; - rowdata = ExecFetchSlotHeapTupleDatum(slot); + tup = heap_form_tuple(cstate->tupDesc, + slot->tts_values, + slot->tts_isnull); + rowdata = HeapTupleGetDatum(tup); + } + else + { + rowdata = ExecFetchSlotHeapTupleDatum(slot); + } cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-09 03:48 jian he <[email protected]> parent: Andrew Dunstan <[email protected]> 0 siblings, 2 replies; 22+ messages in thread From: jian he @ 2026-03-09 03:48 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan <[email protected]> wrote: > > Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? > How about something like this?: > > - * Full table or query without column list. Ensure the slot uses > - * cstate->tupDesc so that the datum is stamped with the right type; > - * for queries output type is RECORDOID this must be the blessed > - * descriptor so that composite_to_json can look it up via > - * lookup_rowtype_tupdesc. > + * Full table or query without column list. For queries, the slot's > + * TupleDesc may carry RECORDOID, which is not registered in the > type > + * cache and would cause composite_to_json's lookup_rowtype_tupdesc > + * call to fail. Build a HeapTuple stamped with the blessed > + * descriptor so the type can be looked up correctly. > */ > if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == > RECORDOID) > - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; > + { > + HeapTuple tup; > > - rowdata = ExecFetchSlotHeapTupleDatum(slot); > + tup = heap_form_tuple(cstate->tupDesc, > + slot->tts_values, > + slot->tts_isnull); > + rowdata = HeapTupleGetDatum(tup); > + } > + else > + { > + rowdata = ExecFetchSlotHeapTupleDatum(slot); > + } > This is better. I've tried to get rid of json_projvalues and json_projnulls. Just using heap_form_tuple, but it won't work. I incorporated the v28-0004 COPY column list into v9-0002. With this patch set, we added four fields to the struct CopyToStateData. + StringInfo json_buf; /* reusable buffer for JSON output, + * initialized in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for JSON output; for a column + * list this is a projected descriptor */ + Datum *json_projvalues; /* pre-allocated projection values, or + * NULL */ + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ Using the script in https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40ma... I tested it again on macOS and Linux, and there are no regressions for COPY TO with the TEXT and CSV formats. -- jian https://www.enterprisedb.com/ Attachments: [text/x-patch] v29-0002-json-format-for-COPY-TO.patch (29.3K, 2-v29-0002-json-format-for-COPY-TO.patch) download | inline diff: From e9fdfae2f0829b2af2e5ee4047230ec124bd72b9 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Mar 2026 10:21:32 +0800 Subject: [PATCH v29 2/3] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a single JSON object or a stream of JSON objects. The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV parsing or formatting options, including: - HEADER - DEFAULT - NULL - DELIMITER - FORCE QUOTE / FORCE NOT NULL Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Author: Andrew Dunstan <[email protected]>, Reviewed-by: "Andrey M. Borodin" <[email protected]>, Reviewed-by: Dean Rasheed <[email protected]>, Reviewed-by: Daniel Verite <[email protected]>, Reviewed-by: Davin Shearer <[email protected]>, Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 ++- src/backend/commands/copy.c | 49 ++++++--- src/backend/commands/copyto.c | 161 +++++++++++++++++++++++++++-- src/backend/parser/gram.y | 8 ++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 146 ++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 88 ++++++++++++++++ 10 files changed, 444 insertions(+), 31 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29c121c7f08 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index b0ee91fc9c1..9d8d8318957 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -85,6 +87,13 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + StringInfo json_buf; /* reusable buffer for JSON output, + * initialized in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for JSON output; for a column + * list this is a projected descriptor */ + Datum *json_projvalues; /* pre-allocated projection values, or + * NULL */ + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -131,6 +140,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,9 +159,6 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. - * - * CSV and text formats share the same TextLike routines except for the - * one-row callback. */ /* text format */ @@ -170,6 +177,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -186,12 +201,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -210,6 +227,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -232,7 +251,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -305,13 +324,79 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + resetStringInfo(cstate->json_buf); + + if (cstate->json_projvalues != NULL) + { + /* + * Column list case: project selected column values into sequential + * positions matching the custom TupleDesc, then form a new tuple. + */ + HeapTuple tup; + int i = 0; + + foreach_int(attnum, cstate->attnumlist) + { + cstate->json_projvalues[i] = slot->tts_values[attnum - 1]; + cstate->json_projnulls[i] = slot->tts_isnull[attnum - 1]; + i++; + } + + tup = heap_form_tuple(cstate->tupDesc, + cstate->json_projvalues, + cstate->json_projnulls); + + /* + * heap_form_tuple already stamps the datum-length, type-id, and + * type-mod fields on t_data, so we can use it directly as a composite + * Datum without the extra pallocmemcpy that heap_copy_tuple_as_datum + * would do. Any TOAST pointers in the projected values will be + * detoasted by the per-column output functions called from + * composite_to_json. + */ + rowdata = HeapTupleGetDatum(tup); + } + else + { + /* + * Full table or query without column list. For queries, the slot's + * TupleDesc may carry RECORDOID, which is not registered in the type + * cache and would cause composite_to_json's lookup_rowtype_tupdesc + * call to fail. Build a HeapTuple stamped with the blessed + * descriptor so the type can be looked up correctly. + */ + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) + { + HeapTuple tup = heap_form_tuple(cstate->tupDesc, + slot->tts_values, + slot->tts_isnull); + + rowdata = HeapTupleGetDatum(tup); + } + else + rowdata = ExecFetchSlotHeapTupleDatum(slot); + } + + composite_to_json(rowdata, cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -403,9 +488,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -507,7 +606,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -750,6 +849,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = RelationGetDescr(cstate->rel); cstate->partitions = children; + cstate->tupDesc = tupDesc; } else { @@ -886,11 +986,52 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; + tupDesc = BlessTupleDesc(tupDesc); + cstate->tupDesc = tupDesc; } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* Set up JSON-specific state */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + cstate->json_buf = makeStringInfo(); + + if (attnamelist != NIL && rel) + { + int natts = list_length(cstate->attnumlist); + TupleDesc resultDesc; + + /* + * Build a TupleDesc describing only the selected columns so that + * composite_to_json() emits the right column names and types. + */ + resultDesc = CreateTemplateTupleDesc(natts); + + foreach_int(attnum, cstate->attnumlist) + { + Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); + + TupleDescInitEntry(resultDesc, + foreach_current_index(attnum) + 1, + NameStr(attr->attname), + attr->atttypid, + attr->atttypmod, + attr->attndims); + } + + cstate->tupDesc = BlessTupleDesc(resultDesc); + + /* + * * Pre-allocate arrays for projecting selected column values + * into sequential positions matching the custom TupleDesc. + */ + cstate->json_projvalues = palloc_array(Datum, natts); + cstate->json_projnulls = palloc_array(bool, natts); + } + } + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 9cbe8eafc45..136fd19b854 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3612,6 +3612,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3694,6 +3698,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 6484c6a3dd4..bb82bdbcc48 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..9b667544905 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,152 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +{"?column?":1} +{"?column?":2} +copy (select 1 as foo union all select 2) to stdout with (format json); +{"foo":1} +{"foo":2} +copy (values (1), (2)) TO stdout with (format json); +{"column1":1} +{"column1":2} +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy (select * from copytest) to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest to stdout (format json, reject_limit 1); +ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE +copy copytest from stdin(format json); +ERROR: COPY JSON mode cannot be used with COPY FROM +-- all of the above should yield error +-- column list with json format +copy copytest (style, test, filler) to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); +-- full table +copy copyjsontest_types to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2},"arr":[1,2,3],"n":3.14,"b":true,"ts":"2024-01-15T10:30:00","t":"hello"} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}},"arr":[4,5],"n":-99.99,"b":false,"ts":"2024-06-30T23:59:59","t":"world"} +{"id":3,"js":null,"jsb":null,"arr":[],"n":null,"b":null,"ts":null,"t":null} +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2}} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}}} +{"id":3,"js":null,"jsb":null} +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +{"id":1,"arr":[1,2,3],"n":3.14,"b":true} +{"id":2,"arr":[4,5],"n":-99.99,"b":false} +{"id":3,"arr":[],"n":null,"b":null} +copy copyjsontest_types (jsb, t) to stdout (format json); +{"jsb":{"b": 2},"t":"hello"} +{"jsb":{"nested": {"k": "v"}},"t":"world"} +{"jsb":null,"t":null} +copy copyjsontest_types (id, ts) to stdout (format json); +{"id":1,"ts":"2024-01-15T10:30:00"} +{"id":2,"ts":"2024-06-30T23:59:59"} +{"id":3,"ts":null} +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +{"js":{"a":1}} +{"js":[1,null,"x"]} +{"js":null} +copy copyjsontest_types (jsb) to stdout (format json); +{"jsb":{"b": 2}} +{"jsb":{"nested": {"k": "v"}}} +{"jsb":null} +drop table copyjsontest_types; +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..404f4321085 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,94 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +copy (select 1 as foo union all select 2) to stdout with (format json); +copy (values (1), (2)) TO stdout with (format json); +copy copytest to stdout json; +copy copytest to stdout (format json); +copy (select * from copytest) to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest to stdout (format json, reject_limit 1); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- column list with json format +copy copytest (style, test, filler) to stdout (format json); + +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); + +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); + +-- full table +copy copyjsontest_types to stdout (format json); + +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +copy copyjsontest_types (jsb, t) to stdout (format json); +copy copyjsontest_types (id, ts) to stdout (format json); + +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +copy copyjsontest_types (jsb) to stdout (format json); + +drop table copyjsontest_types; + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.34.1 [text/x-patch] v29-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 3-v29-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From cc747df3929913078b85dd5fbee5a852aa7d0d53 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Wed, 21 Jan 2026 18:38:24 +0800 Subject: [PATCH v29 1/3] introduce CopyFormat refactor CopyFormatOptions Currently, COPY command format is determined by two booleans (binary, csv_mode) fields in CopyFormatOptions This approach, while functional, isn't ideal for future other implement other format. To simplify adding new formats, we've introduced an enum CopyFormat. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. The CopyFormat enum was originally contributed by Joel Jacobson <[email protected]>, later refactored by Jian He to address various issues. discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2f42f55e229..4d927410159 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 84c8809a889..5f0c551e7ec 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -173,7 +173,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -751,7 +751,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -778,7 +778,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index d6ef7275a64..b0ee91fc9c1 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -182,9 +182,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -221,7 +221,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -398,7 +398,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 3250564d4ff..520cdd36800 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.34.1 [text/x-patch] v29-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (12.3K, 4-v29-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 510f1f0435bc74b80575b0b8af2134817fedfa05 Mon Sep 17 00:00:00 2001 From: jian he <[email protected]> Date: Mon, 9 Mar 2026 10:27:37 +0800 Subject: [PATCH v29 3/3] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]>, Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 ++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 41 ++++++++++++++++++++++++++++-- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 33 ++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 10 ++++++++ 7 files changed, 127 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29c121c7f08..84254d46a67 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 9d8d8318957..85ca7c947f3 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -87,6 +87,7 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, * initialized in BeginCopyTo */ TupleDesc tupDesc; /* Descriptor for JSON output; for a column @@ -141,6 +142,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -182,7 +184,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -248,6 +250,18 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } + } } /* @@ -324,13 +338,24 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) @@ -392,6 +417,18 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) composite_to_json(rowdata, cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); CopySendTextLikeEndOfRow(cstate); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index bb82bdbcc48..00458cfb4bc 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 9b667544905..bcf45845b61 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -83,6 +83,16 @@ copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} +copy (select 1 union all select 2) to stdout with (format json, force_array true); +[ + {"?column?":1} +,{"?column?":2} +] +copy (values (1), (2)) TO stdout with (format json, force_array true); +[ + {"column1":1} +,{"column1":2} +] copy copytest to stdout json; {"style":"DOS","test":"abc\r\ndef","filler":1} {"style":"Unix","test":"abc\ndef","filler":2} @@ -134,6 +144,29 @@ copy copytest (style, test, filler) to stdout (format json); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest(style, test) to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef"} +,{"style":"Unix","test":"abc\ndef"} +,{"style":"Mac","test":"abc\rdef"} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb"} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} -- column list with diverse data types create temp table copyjsontest_types ( id int, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 404f4321085..bc12ac879ef 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -86,6 +86,8 @@ copy copytest3 to stdout csv header; copy (select 1 union all select 2) to stdout with (format json); copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); +copy (select 1 union all select 2) to stdout with (format json, force_array true); +copy (values (1), (2)) TO stdout with (format json, force_array true); copy copytest to stdout json; copy copytest to stdout (format json); copy (select * from copytest) to stdout (format json); @@ -109,6 +111,14 @@ copy copytest from stdin(format json); -- column list with json format copy copytest (style, test, filler) to stdout (format json); +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest(style, test) to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + -- column list with diverse data types create temp table copyjsontest_types ( id int, -- 2.34.1 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-09 12:23 Andrew Dunstan <[email protected]> parent: jian he <[email protected]> 1 sibling, 0 replies; 22+ messages in thread From: Andrew Dunstan @ 2026-03-09 12:23 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-03-08 Su 11:48 PM, jian he wrote: > On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan<[email protected]> wrote: >> Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? >> How about something like this?: >> >> - * Full table or query without column list. Ensure the slot uses >> - * cstate->tupDesc so that the datum is stamped with the right type; >> - * for queries output type is RECORDOID this must be the blessed >> - * descriptor so that composite_to_json can look it up via >> - * lookup_rowtype_tupdesc. >> + * Full table or query without column list. For queries, the slot's >> + * TupleDesc may carry RECORDOID, which is not registered in the >> type >> + * cache and would cause composite_to_json's lookup_rowtype_tupdesc >> + * call to fail. Build a HeapTuple stamped with the blessed >> + * descriptor so the type can be looked up correctly. >> */ >> if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == >> RECORDOID) >> - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; >> + { >> + HeapTuple tup; >> >> - rowdata = ExecFetchSlotHeapTupleDatum(slot); >> + tup = heap_form_tuple(cstate->tupDesc, >> + slot->tts_values, >> + slot->tts_isnull); >> + rowdata = HeapTupleGetDatum(tup); >> + } >> + else >> + { >> + rowdata = ExecFetchSlotHeapTupleDatum(slot); >> + } >> > This is better. I've tried to get rid of json_projvalues and json_projnulls. > Just using heap_form_tuple, but it won't work. > > I incorporated the v28-0004 COPY column list into v9-0002. > With this patch set, we added four fields to the struct CopyToStateData. > > + StringInfo json_buf; /* reusable buffer for JSON output, > + * initialized in BeginCopyTo */ > + TupleDesc tupDesc; /* Descriptor for JSON output; for a column > + * list this is a projected descriptor */ > + Datum *json_projvalues; /* pre-allocated projection values, or > + * NULL */ > + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ > > Using the script in > https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40ma... > I tested it again on macOS and Linux, and there are no regressions for > COPY TO with the TEXT and CSV formats. > OK, I think we're really close now. Here is a tiny fixup patch that fixes an error message and a comment, and adds a missing test case. cheers andrew -- Andrew Dunstan EDB:https://www.enterprisedb.com diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 84254d46a67..c11c2eb48fa 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -972,8 +972,8 @@ ProcessCopyOptions(ParseState *pstate, /* Check json format */ if (opts_out->format == COPY_FORMAT_JSON && is_from) ereport(ERROR, - errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("COPY %s is not supported for %s", "FORMAT JSON", "COPY FROM")); if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) ereport(ERROR, diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 85ca7c947f3..de280ad8612 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -1061,8 +1061,8 @@ BeginCopyTo(ParseState *pstate, cstate->tupDesc = BlessTupleDesc(resultDesc); /* - * * Pre-allocate arrays for projecting selected column values - * into sequential positions matching the custom TupleDesc. + * Pre-allocate arrays for projecting selected column values into + * sequential positions matching the custom TupleDesc. */ cstate->json_projvalues = palloc_array(Datum, natts); cstate->json_projnulls = palloc_array(bool, natts); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index bcf45845b61..3e566b96183 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -136,7 +136,7 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest to stdout (format json, reject_limit 1); ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE copy copytest from stdin(format json); -ERROR: COPY JSON mode cannot be used with COPY FROM +ERROR: COPY FORMAT JSON is not supported for COPY FROM -- all of the above should yield error -- column list with json format copy copytest (style, test, filler) to stdout (format json); @@ -167,6 +167,10 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- empty result set with force_array +copy (select 1 where false) to stdout (format json, force_array true); +[ +] -- column list with diverse data types create temp table copyjsontest_types ( id int, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index bc12ac879ef..520934271ec 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -119,6 +119,9 @@ copy copytest to stdout (format json, force_array); copy copytest(style, test) to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- empty result set with force_array +copy (select 1 where false) to stdout (format json, force_array true); + -- column list with diverse data types create temp table copyjsontest_types ( id int, Attachments: [text/plain] copy-json-fixes.patch.nocfbot (3.0K, 3-copy-json-fixes.patch.nocfbot) download | inline diff: diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 84254d46a67..c11c2eb48fa 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -972,8 +972,8 @@ ProcessCopyOptions(ParseState *pstate, /* Check json format */ if (opts_out->format == COPY_FORMAT_JSON && is_from) ereport(ERROR, - errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY %s mode cannot be used with %s", "JSON", "COPY FROM")); + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("COPY %s is not supported for %s", "FORMAT JSON", "COPY FROM")); if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) ereport(ERROR, diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 85ca7c947f3..de280ad8612 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -1061,8 +1061,8 @@ BeginCopyTo(ParseState *pstate, cstate->tupDesc = BlessTupleDesc(resultDesc); /* - * * Pre-allocate arrays for projecting selected column values - * into sequential positions matching the custom TupleDesc. + * Pre-allocate arrays for projecting selected column values into + * sequential positions matching the custom TupleDesc. */ cstate->json_projvalues = palloc_array(Datum, natts); cstate->json_projnulls = palloc_array(bool, natts); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index bcf45845b61..3e566b96183 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -136,7 +136,7 @@ LINE 1: copy copytest to stdout (format json, on_error ignore); copy copytest to stdout (format json, reject_limit 1); ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE copy copytest from stdin(format json); -ERROR: COPY JSON mode cannot be used with COPY FROM +ERROR: COPY FORMAT JSON is not supported for COPY FROM -- all of the above should yield error -- column list with json format copy copytest (style, test, filler) to stdout (format json); @@ -167,6 +167,10 @@ copy copytest to stdout (format json, force_array false); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- empty result set with force_array +copy (select 1 where false) to stdout (format json, force_array true); +[ +] -- column list with diverse data types create temp table copyjsontest_types ( id int, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index bc12ac879ef..520934271ec 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -119,6 +119,9 @@ copy copytest to stdout (format json, force_array); copy copytest(style, test) to stdout (format json, force_array true); copy copytest to stdout (format json, force_array false); +-- empty result set with force_array +copy (select 1 where false) to stdout (format json, force_array true); + -- column list with diverse data types create temp table copyjsontest_types ( id int, ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-16 18:24 Masahiko Sawada <[email protected]> parent: jian he <[email protected]> 1 sibling, 1 reply; 22+ messages in thread From: Masahiko Sawada @ 2026-03-16 18:24 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Sun, Mar 8, 2026 at 8:49 PM jian he <[email protected]> wrote: > > On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan <[email protected]> wrote: > > > > Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? > > How about something like this?: > > > > - * Full table or query without column list. Ensure the slot uses > > - * cstate->tupDesc so that the datum is stamped with the right type; > > - * for queries output type is RECORDOID this must be the blessed > > - * descriptor so that composite_to_json can look it up via > > - * lookup_rowtype_tupdesc. > > + * Full table or query without column list. For queries, the slot's > > + * TupleDesc may carry RECORDOID, which is not registered in the > > type > > + * cache and would cause composite_to_json's lookup_rowtype_tupdesc > > + * call to fail. Build a HeapTuple stamped with the blessed > > + * descriptor so the type can be looked up correctly. > > */ > > if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == > > RECORDOID) > > - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; > > + { > > + HeapTuple tup; > > > > - rowdata = ExecFetchSlotHeapTupleDatum(slot); > > + tup = heap_form_tuple(cstate->tupDesc, > > + slot->tts_values, > > + slot->tts_isnull); > > + rowdata = HeapTupleGetDatum(tup); > > + } > > + else > > + { > > + rowdata = ExecFetchSlotHeapTupleDatum(slot); > > + } > > > This is better. I've tried to get rid of json_projvalues and json_projnulls. > Just using heap_form_tuple, but it won't work. > > I incorporated the v28-0004 COPY column list into v9-0002. > With this patch set, we added four fields to the struct CopyToStateData. > > + StringInfo json_buf; /* reusable buffer for JSON output, > + * initialized in BeginCopyTo */ > + TupleDesc tupDesc; /* Descriptor for JSON output; for a column > + * list this is a projected descriptor */ > + Datum *json_projvalues; /* pre-allocated projection values, or > + * NULL */ > + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ > > Using the script in > https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40ma... > I tested it again on macOS and Linux, and there are no regressions for > COPY TO with the TEXT and CSV formats. I've reviewed the patch and have some comments: --- I got a SEGV in the following scenario: postgres(1:1197708)=# create table test (a int, b text, c jsonb); CREATE TABLE postgres(1:1197708)=# copy test(a, b) to stdout with (format 'json' ); TRAP: failed Assert("tupdesc->firstNonCachedOffsetAttr >= 0"), File: "execTuples.c", Line: 2328, PID: 1197708 postgres: masahiko postgres [local] COPY(ExceptionalCondition+0x9e) [0xbebe48] postgres: masahiko postgres [local] COPY(BlessTupleDesc+0x2b) [0x729b50] postgres: masahiko postgres [local] COPY(BeginCopyTo+0xc94) [0x637bdf] postgres: masahiko postgres [local] COPY(DoCopy+0xb68) [0x62afbc] postgres: masahiko postgres [local] COPY(standard_ProcessUtility+0xa22) [0xa0ba48] postgres: masahiko postgres [local] COPY(ProcessUtility+0x10e) [0xa0b01f] postgres: masahiko postgres [local] COPY() [0xa09872] postgres: masahiko postgres [local] COPY() [0xa09acf] postgres: masahiko postgres [local] COPY(PortalRun+0x2c8) [0xa0901d] postgres: masahiko postgres [local] COPY() [0xa02055] postgres: masahiko postgres [local] COPY(PostgresMain+0xaf1) [0xa0724e] postgres: masahiko postgres [local] COPY() [0x9fdab9] postgres: masahiko postgres [local] COPY(postmaster_child_launch+0x165) [0x905378] postgres: masahiko postgres [local] COPY() [0x90b600] postgres: masahiko postgres [local] COPY() [0x908e6a] postgres: masahiko postgres [local] COPY(PostmasterMain+0x14fe) [0x90880c] postgres: masahiko postgres [local] COPY(main+0x340) [0x7a1f9c] It seems to forget to call TupleDescFinalize(). And I think we need some regression tests for this case. --- + if (cstate->opts.format == COPY_FORMAT_JSON) + { + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } + } We can conjunct the two if statement conditions. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-16 20:59 Andrew Dunstan <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 2 replies; 22+ messages in thread From: Andrew Dunstan @ 2026-03-16 20:59 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; jian he <[email protected]>; +Cc: Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-03-16 Mo 2:24 PM, Masahiko Sawada wrote: > On Sun, Mar 8, 2026 at 8:49 PM jian he<[email protected]> wrote: >> On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan<[email protected]> wrote: >>> Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? >>> How about something like this?: >>> >>> - * Full table or query without column list. Ensure the slot uses >>> - * cstate->tupDesc so that the datum is stamped with the right type; >>> - * for queries output type is RECORDOID this must be the blessed >>> - * descriptor so that composite_to_json can look it up via >>> - * lookup_rowtype_tupdesc. >>> + * Full table or query without column list. For queries, the slot's >>> + * TupleDesc may carry RECORDOID, which is not registered in the >>> type >>> + * cache and would cause composite_to_json's lookup_rowtype_tupdesc >>> + * call to fail. Build a HeapTuple stamped with the blessed >>> + * descriptor so the type can be looked up correctly. >>> */ >>> if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == >>> RECORDOID) >>> - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; >>> + { >>> + HeapTuple tup; >>> >>> - rowdata = ExecFetchSlotHeapTupleDatum(slot); >>> + tup = heap_form_tuple(cstate->tupDesc, >>> + slot->tts_values, >>> + slot->tts_isnull); >>> + rowdata = HeapTupleGetDatum(tup); >>> + } >>> + else >>> + { >>> + rowdata = ExecFetchSlotHeapTupleDatum(slot); >>> + } >>> >> This is better. I've tried to get rid of json_projvalues and json_projnulls. >> Just using heap_form_tuple, but it won't work. >> >> I incorporated the v28-0004 COPY column list into v9-0002. >> With this patch set, we added four fields to the struct CopyToStateData. >> >> + StringInfo json_buf; /* reusable buffer for JSON output, >> + * initialized in BeginCopyTo */ >> + TupleDesc tupDesc; /* Descriptor for JSON output; for a column >> + * list this is a projected descriptor */ >> + Datum *json_projvalues; /* pre-allocated projection values, or >> + * NULL */ >> + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ >> >> Using the script in >> https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40ma... >> I tested it again on macOS and Linux, and there are no regressions for >> COPY TO with the TEXT and CSV formats. > I've reviewed the patch and have some comments: > > --- > I got a SEGV in the following scenario: > > postgres(1:1197708)=# create table test (a int, b text, c jsonb); > CREATE TABLE > postgres(1:1197708)=# copy test(a, b) to stdout with (format 'json' ); > TRAP: failed Assert("tupdesc->firstNonCachedOffsetAttr >= 0"), File: > "execTuples.c", Line: 2328, PID: 1197708 > postgres: masahiko postgres [local] COPY(ExceptionalCondition+0x9e) [0xbebe48] > postgres: masahiko postgres [local] COPY(BlessTupleDesc+0x2b) [0x729b50] > postgres: masahiko postgres [local] COPY(BeginCopyTo+0xc94) [0x637bdf] > postgres: masahiko postgres [local] COPY(DoCopy+0xb68) [0x62afbc] > postgres: masahiko postgres [local] > COPY(standard_ProcessUtility+0xa22) [0xa0ba48] > postgres: masahiko postgres [local] COPY(ProcessUtility+0x10e) [0xa0b01f] > postgres: masahiko postgres [local] COPY() [0xa09872] > postgres: masahiko postgres [local] COPY() [0xa09acf] > postgres: masahiko postgres [local] COPY(PortalRun+0x2c8) [0xa0901d] > postgres: masahiko postgres [local] COPY() [0xa02055] > postgres: masahiko postgres [local] COPY(PostgresMain+0xaf1) [0xa0724e] > postgres: masahiko postgres [local] COPY() [0x9fdab9] > postgres: masahiko postgres [local] > COPY(postmaster_child_launch+0x165) [0x905378] > postgres: masahiko postgres [local] COPY() [0x90b600] > postgres: masahiko postgres [local] COPY() [0x908e6a] > postgres: masahiko postgres [local] COPY(PostmasterMain+0x14fe) [0x90880c] > postgres: masahiko postgres [local] COPY(main+0x340) [0x7a1f9c] > > It seems to forget to call TupleDescFinalize(). And I think we need > some regression tests for this case. > > --- > + if (cstate->opts.format == COPY_FORMAT_JSON) > + { > + /* > + * If FORCE_ARRAY has been specified, send the opening bracket. > + */ > + if (cstate->opts.force_array) > + { > + CopySendChar(cstate, '['); > + CopySendTextLikeEndOfRow(cstate); > + } > + } > > We can conjunct the two if statement conditions. > Here's a v30 set that I hope fixes these issues. cheers andrew -- Andrew Dunstan EDB:https://www.enterprisedb.com Attachments: [text/x-patch] v30-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (13.1K, 3-v30-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch) download | inline diff: From f05872732a3654362d916354a2edd071aa131099 Mon Sep 17 00:00:00 2001 From: Andrew Dunstan <[email protected]> Date: Mon, 16 Mar 2026 16:49:01 -0400 Subject: [PATCH v30 1/3] introduce CopyFormat, refactor CopyFormatOptions Currently, the COPY command format is determined by two boolean fields (binary, csv_mode) in CopyFormatOptions. This approach, while functional, isn't ideal for implementing other formats in the future. To simplify adding new formats, introduce a CopyFormat enum. This makes the code cleaner and more maintainable, allowing for easier integration of additional formats down the line. Author: Joel Jacobson <[email protected]> Author: jian he <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> Discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com Discussion: https://postgr.es/m/[email protected] --- src/backend/commands/copy.c | 50 +++++++++++++++------------- src/backend/commands/copyfrom.c | 6 ++-- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 8 ++--- src/include/commands/copy.h | 13 ++++++-- src/tools/pgindent/typedefs.list | 1 + 6 files changed, 49 insertions(+), 36 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 63b86802ba2..2f46be516f2 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -576,6 +576,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out = palloc0_object(CopyFormatOptions); opts_out->file_encoding = -1; + /* default format */ + opts_out->format = COPY_FORMAT_TEXT; /* Extract options from the statement node tree */ foreach(option, options) @@ -590,11 +592,11 @@ ProcessCopyOptions(ParseState *pstate, errorConflictingDefElem(defel, pstate); format_specified = true; if (strcmp(fmt, "text") == 0) - /* default format */ ; + opts_out->format = COPY_FORMAT_TEXT; else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; + opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; + opts_out->format = COPY_FORMAT_BINARY; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -754,31 +756,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = (opts_out->format == COPY_FORMAT_CSV) ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = (opts_out->format == COPY_FORMAT_CSV) ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (opts_out->format == COPY_FORMAT_CSV) { if (!opts_out->quote) opts_out->quote = "\""; @@ -826,7 +828,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (opts_out->format != COPY_FORMAT_CSV && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -834,43 +836,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (opts_out->format == COPY_FORMAT_CSV && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (opts_out->format != COPY_FORMAT_CSV && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (opts_out->format == COPY_FORMAT_CSV && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -884,8 +886,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -900,8 +902,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (opts_out->format != COPY_FORMAT_CSV && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -925,7 +927,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -961,7 +963,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (opts_out->format == COPY_FORMAT_CSV && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -978,7 +980,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (opts_out->format == COPY_FORMAT_BINARY && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 95f6cb416a9..a7fe29a363a 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -156,9 +156,9 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyFromRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyFromRoutineBinary; /* default is text */ @@ -262,7 +262,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.format == COPY_FORMAT_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 00ee4154b8b..200d7bc79cd 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -174,7 +174,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -752,7 +752,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.format == COPY_FORMAT_CSV); } /* @@ -779,7 +779,8 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done = false; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.format == COPY_FORMAT_TEXT || + cstate->opts.format == COPY_FORMAT_CSV); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line != COPY_HEADER_FALSE) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index d6ef7275a64..b0ee91fc9c1 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -182,9 +182,9 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (opts->format == COPY_FORMAT_CSV) return &CopyToRoutineCSV; - else if (opts->binary) + else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; /* default is text */ @@ -221,7 +221,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.format == COPY_FORMAT_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -398,7 +398,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.format == COPY_FORMAT_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 877202af67b..2430fb0b2e5 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -49,6 +49,16 @@ typedef enum CopyLogVerbosityChoice COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ } CopyLogVerbosityChoice; +/* + * Represents the format of the COPY operation. + */ +typedef enum CopyFormat +{ + COPY_FORMAT_TEXT = 0, + COPY_FORMAT_BINARY, + COPY_FORMAT_CSV, +} CopyFormat; + /* * A struct to hold COPY options, in a parsed form. All of these are related * to formatting, except for 'freeze', which doesn't really belong here, but @@ -59,9 +69,8 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ + CopyFormat format; /* format of the COPY operation */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ int header_line; /* number of lines to skip or COPY_HEADER_XXX * value (see the above) */ char *null_print; /* NULL marker string (server encoding!) */ diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index ec8513d90b5..3b2fcebcb54 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -528,6 +528,7 @@ ConversionLocation ConvertRowtypeExpr CookedConstraint CopyDest +CopyFormat CopyFormatOptions CopyFromRoutine CopyFromState -- 2.43.0 [text/x-patch] v30-0002-json-format-for-COPY-TO.patch (29.5K, 4-v30-0002-json-format-for-COPY-TO.patch) download | inline diff: From 9d13f458dea5c3914705b06c0826c393d02cebcf Mon Sep 17 00:00:00 2001 From: Andrew Dunstan <[email protected]> Date: Mon, 16 Mar 2026 16:50:24 -0400 Subject: [PATCH v30 2/3] json format for COPY TO This introduces the JSON format option for the COPY TO command, allowing users to export query results or table data directly as a stream of JSON objects (one per line, NDJSON style). The JSON format is currently supported only for COPY TO operations; it is not available for COPY FROM. JSON format is incompatible with some standard text/CSV formatting options, including HEADER, DEFAULT, NULL, DELIMITER, FORCE QUOTE, FORCE NOT NULL, and FORCE NULL. Column list support is included: when a column list is specified, only the named columns are emitted in each JSON object. Regression tests covering valid JSON exports and error handling for incompatible options have been added to src/test/regress/sql/copy.sql. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Andrey M. Borodin <[email protected]> Reviewed-by: Dean Rasheed <[email protected]> Reviewed-by: Daniel Verite <[email protected]> Reviewed-by: Davin Shearer <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]> Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> Discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com Discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 13 ++- src/backend/commands/copy.c | 49 ++++++--- src/backend/commands/copyto.c | 162 +++++++++++++++++++++++++++-- src/backend/parser/gram.y | 8 ++ src/backend/utils/adt/json.c | 5 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/include/utils/json.h | 2 + src/test/regress/expected/copy.out | 146 ++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 88 ++++++++++++++++ 10 files changed, 445 insertions(+), 31 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 0ad890ef95f..75f55bbf6f8 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -228,10 +228,15 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable Selects the data format to be read or written: <literal>text</literal>, <literal>csv</literal> (Comma Separated Values), + <literal>json</literal> (JavaScript Object Notation), or <literal>binary</literal>. The default is <literal>text</literal>. See <xref linkend="sql-copy-file-formats"/> below for details. </para> + <para> + The <literal>json</literal> option is allowed only in + <command>COPY TO</command>. + </para> </listitem> </varlistentry> @@ -266,7 +271,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable (line) of the file. The default is a tab character in text format, a comma in <literal>CSV</literal> format. This must be a single one-byte character. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -280,7 +285,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable string in <literal>CSV</literal> format. You might prefer an empty string even in text format for cases where you don't want to distinguish nulls from empty strings. - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> <note> @@ -303,7 +308,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable is found in the input file, the default value of the corresponding column will be used. This option is allowed only in <command>COPY FROM</command>, and only when - not using <literal>binary</literal> format. + not using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> @@ -330,7 +335,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable <command>COPY FROM</command> commands. </para> <para> - This option is not allowed when using <literal>binary</literal> format. + This option is not allowed when using <literal>binary</literal> or <literal>json</literal> format. </para> </listitem> </varlistentry> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 2f46be516f2..29e22d91ecd 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -597,6 +597,8 @@ ProcessCopyOptions(ParseState *pstate, opts_out->format = COPY_FORMAT_CSV; else if (strcmp(fmt, "binary") == 0) opts_out->format = COPY_FORMAT_BINARY; + else if (strcmp(fmt, "json") == 0) + opts_out->format = COPY_FORMAT_JSON; else ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -756,21 +758,32 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->delim) + if (opts_out->delim && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DELIMITER") + : errmsg("cannot specify %s in JSON mode", "DELIMITER")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->null_print) + if (opts_out->null_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "NULL"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "NULL") + : errmsg("cannot specify %s in JSON mode", "NULL")); - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->default_print) + if (opts_out->default_print && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_SYNTAX_ERROR), - errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); + errcode(ERRCODE_SYNTAX_ERROR), + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "DEFAULT") + : errmsg("cannot specify %s in JSON mode", "DEFAULT")); /* Set defaults for omitted options */ if (!opts_out->delim) @@ -836,11 +849,15 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->format == COPY_FORMAT_BINARY && opts_out->header_line != COPY_HEADER_FALSE) + if (opts_out->header_line != COPY_HEADER_FALSE && + (opts_out->format == COPY_FORMAT_BINARY || + opts_out->format == COPY_FORMAT_JSON)) ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ - errmsg("cannot specify %s in BINARY mode", "HEADER"))); + opts_out->format == COPY_FORMAT_BINARY + ? errmsg("cannot specify %s in BINARY mode", "HEADER") + : errmsg("cannot specify %s in JSON mode", "HEADER")); /* Check quote */ if (opts_out->format != COPY_FORMAT_CSV && opts_out->quote != NULL) @@ -944,6 +961,12 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY %s cannot be used with %s", "FREEZE", "COPY TO"))); + /* Check json format */ + if (opts_out->format == COPY_FORMAT_JSON && is_from) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s is not supported for %s", "FORMAT JSON", "COPY FROM")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index b0ee91fc9c1..ffe2268fbb0 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -26,6 +26,7 @@ #include "executor/execdesc.h" #include "executor/executor.h" #include "executor/tuptable.h" +#include "funcapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -33,6 +34,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/json.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -85,6 +87,13 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + StringInfo json_buf; /* reusable buffer for JSON output, + * initialized in BeginCopyTo */ + TupleDesc tupDesc; /* Descriptor for JSON output; for a column + * list this is a projected descriptor */ + Datum *json_projvalues; /* pre-allocated projection values, or + * NULL */ + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ copy_data_dest_cb data_dest_cb; /* function for writing data */ CopyFormatOptions opts; @@ -131,6 +140,7 @@ static void CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); +static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -149,9 +159,6 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* * COPY TO routines for built-in formats. - * - * CSV and text formats share the same TextLike routines except for the - * one-row callback. */ /* text format */ @@ -170,6 +177,14 @@ static const CopyToRoutine CopyToRoutineCSV = { .CopyToEnd = CopyToTextLikeEnd, }; +/* json format */ +static const CopyToRoutine CopyToRoutineJson = { + .CopyToStart = CopyToTextLikeStart, + .CopyToOutFunc = CopyToTextLikeOutFunc, + .CopyToOneRow = CopyToJsonOneRow, + .CopyToEnd = CopyToTextLikeEnd, +}; + /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { .CopyToStart = CopyToBinaryStart, @@ -186,12 +201,14 @@ CopyToGetRoutine(const CopyFormatOptions *opts) return &CopyToRoutineCSV; else if (opts->format == COPY_FORMAT_BINARY) return &CopyToRoutineBinary; + else if (opts->format == COPY_FORMAT_JSON) + return &CopyToRoutineJson; /* default is text */ return &CopyToRoutineText; } -/* Implementation of the start callback for text and CSV formats */ +/* Implementation of the start callback for text, CSV, and json formats */ static void CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) { @@ -210,6 +227,8 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) ListCell *cur; bool hdr_delim = false; + Assert(cstate->opts.format != COPY_FORMAT_JSON); + foreach(cur, cstate->attnumlist) { int attnum = lfirst_int(cur); @@ -232,7 +251,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) } /* - * Implementation of the outfunc callback for text and CSV formats. Assign + * Implementation of the outfunc callback for text, CSV, and json formats. Assign * the output function data to the given *finfo. */ static void @@ -305,13 +324,79 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text and CSV formats */ +/* Implementation of the end callback for text, CSV, and json formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of per-row callback for json format */ +static void +CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + Datum rowdata; + + resetStringInfo(cstate->json_buf); + + if (cstate->json_projvalues != NULL) + { + /* + * Column list case: project selected column values into sequential + * positions matching the custom TupleDesc, then form a new tuple. + */ + HeapTuple tup; + int i = 0; + + foreach_int(attnum, cstate->attnumlist) + { + cstate->json_projvalues[i] = slot->tts_values[attnum - 1]; + cstate->json_projnulls[i] = slot->tts_isnull[attnum - 1]; + i++; + } + + tup = heap_form_tuple(cstate->tupDesc, + cstate->json_projvalues, + cstate->json_projnulls); + + /* + * heap_form_tuple already stamps the datum-length, type-id, and + * type-mod fields on t_data, so we can use it directly as a composite + * Datum without the extra pallocmemcpy that heap_copy_tuple_as_datum + * would do. Any TOAST pointers in the projected values will be + * detoasted by the per-column output functions called from + * composite_to_json. + */ + rowdata = HeapTupleGetDatum(tup); + } + else + { + /* + * Full table or query without column list. For queries, the slot's + * TupleDesc may carry RECORDOID, which is not registered in the type + * cache and would cause composite_to_json's lookup_rowtype_tupdesc + * call to fail. Build a HeapTuple stamped with the blessed + * descriptor so the type can be looked up correctly. + */ + if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == RECORDOID) + { + HeapTuple tup = heap_form_tuple(cstate->tupDesc, + slot->tts_values, + slot->tts_isnull); + + rowdata = HeapTupleGetDatum(tup); + } + else + rowdata = ExecFetchSlotHeapTupleDatum(slot); + } + + composite_to_json(rowdata, cstate->json_buf, false); + + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); + + CopySendTextLikeEndOfRow(cstate); +} + /* * Implementation of the start callback for binary format. Send a header * for a binary copy. @@ -403,9 +488,23 @@ SendCopyBegin(CopyToState cstate) pq_beginmessage(&buf, PqMsg_CopyOutResponse); pq_sendbyte(&buf, format); /* overall format */ - pq_sendint16(&buf, natts); - for (i = 0; i < natts; i++) - pq_sendint16(&buf, format); /* per-column formats */ + if (cstate->opts.format != COPY_FORMAT_JSON) + { + pq_sendint16(&buf, natts); + for (i = 0; i < natts; i++) + pq_sendint16(&buf, format); /* per-column formats */ + } + else + { + /* + * For JSON format, report one text-format column. Each CopyData + * message contains one complete JSON object, not individual column + * values, so the per-column count is always 1. + */ + pq_sendint16(&buf, 1); + pq_sendint16(&buf, 0); + } + pq_endmessage(&buf); cstate->copy_dest = COPY_FRONTEND; } @@ -507,7 +606,7 @@ CopySendEndOfRow(CopyToState cstate) } /* - * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the + * Wrapper function of CopySendEndOfRow for text, CSV, and json formats. Sends the * line termination and do common appropriate things for the end of row. */ static inline void @@ -750,6 +849,7 @@ BeginCopyTo(ParseState *pstate, tupDesc = RelationGetDescr(cstate->rel); cstate->partitions = children; + cstate->tupDesc = tupDesc; } else { @@ -886,11 +986,53 @@ BeginCopyTo(ParseState *pstate, ExecutorStart(cstate->queryDesc, 0); tupDesc = cstate->queryDesc->tupDesc; + tupDesc = BlessTupleDesc(tupDesc); + cstate->tupDesc = tupDesc; } /* Generate or convert list of attributes to process */ cstate->attnumlist = CopyGetAttnums(tupDesc, cstate->rel, attnamelist); + /* Set up JSON-specific state */ + if (cstate->opts.format == COPY_FORMAT_JSON) + { + cstate->json_buf = makeStringInfo(); + + if (attnamelist != NIL && rel) + { + int natts = list_length(cstate->attnumlist); + TupleDesc resultDesc; + + /* + * Build a TupleDesc describing only the selected columns so that + * composite_to_json() emits the right column names and types. + */ + resultDesc = CreateTemplateTupleDesc(natts); + + foreach_int(attnum, cstate->attnumlist) + { + Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1); + + TupleDescInitEntry(resultDesc, + foreach_current_index(attnum) + 1, + NameStr(attr->attname), + attr->atttypid, + attr->atttypmod, + attr->attndims); + } + + TupleDescFinalize(resultDesc); + cstate->tupDesc = BlessTupleDesc(resultDesc); + + /* + * Pre-allocate arrays for projecting selected column values + * into sequential positions matching the custom TupleDesc. + */ + cstate->json_projvalues = palloc_array(Datum, natts); + cstate->json_projnulls = palloc_array(bool, natts); + } + } + num_phys_attrs = tupDesc->natts; /* Convert FORCE_QUOTE name list to per-column flags, check validity */ diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f01f5734fe9..c01b9fc3997 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -3617,6 +3617,10 @@ copy_opt_item: { $$ = makeDefElem("format", (Node *) makeString("csv"), @1); } + | JSON + { + $$ = makeDefElem("format", (Node *) makeString("json"), @1); + } | HEADER_P { $$ = makeDefElem("header", (Node *) makeBoolean(true), @1); @@ -3699,6 +3703,10 @@ copy_generic_opt_elem: { $$ = makeDefElem($1, $2, @1); } + | FORMAT_LA copy_generic_opt_arg + { + $$ = makeDefElem("format", $2, @1); + } ; copy_generic_opt_arg: diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c index 0b161398465..f609d7b9417 100644 --- a/src/backend/utils/adt/json.c +++ b/src/backend/utils/adt/json.c @@ -86,8 +86,6 @@ typedef struct JsonAggState JsonUniqueBuilderState unique_check; } JsonAggState; -static void composite_to_json(Datum composite, StringInfo result, - bool use_line_feeds); static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims, const Datum *vals, const bool *nulls, int *valcount, JsonTypeCategory tcategory, Oid outfuncoid, @@ -517,8 +515,9 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds) /* * Turn a composite / record into JSON. + * Exported so COPY TO can use it. */ -static void +void composite_to_json(Datum composite, StringInfo result, bool use_line_feeds) { HeapTupleHeader td; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 199fc64ddf5..ac36f4591f7 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -3425,7 +3425,7 @@ match_previous_words(int pattern_id, /* Complete COPY <sth> FROM|TO filename WITH (FORMAT */ else if (TailMatches("FORMAT")) - COMPLETE_WITH("binary", "csv", "text"); + COMPLETE_WITH("binary", "csv", "text", "json"); /* Complete COPY <sth> FROM|TO filename WITH (FREEZE */ else if (TailMatches("FREEZE")) diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2430fb0b2e5..2b5bef6738e 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -57,6 +57,7 @@ typedef enum CopyFormat COPY_FORMAT_TEXT = 0, COPY_FORMAT_BINARY, COPY_FORMAT_CSV, + COPY_FORMAT_JSON, } CopyFormat; /* diff --git a/src/include/utils/json.h b/src/include/utils/json.h index f8cc52b1e78..2f4be40518d 100644 --- a/src/include/utils/json.h +++ b/src/include/utils/json.h @@ -17,6 +17,8 @@ #include "lib/stringinfo.h" /* functions in json.c */ +extern void composite_to_json(Datum composite, StringInfo result, + bool use_line_feeds); extern void escape_json(StringInfo buf, const char *str); extern void escape_json_with_len(StringInfo buf, const char *str, int len); extern void escape_json_text(StringInfo buf, const text *txt); diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index d0d563e0fa8..7f2d2e065f6 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -73,6 +73,152 @@ copy copytest3 to stdout csv header; c1,"col with , comma","col with "" quote" 1,a,1 2,b,2 +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +{"?column?":1} +{"?column?":2} +copy (select 1 as foo union all select 2) to stdout with (format json); +{"foo":1} +{"foo":2} +copy (values (1), (2)) TO stdout with (format json); +{"column1":1} +{"column1":2} +copy copytest to stdout json; +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy copytest to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +copy (select * from copytest) to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +ERROR: cannot specify DELIMITER in JSON mode +copy copytest to stdout (format json, null '\N'); +ERROR: cannot specify NULL in JSON mode +copy copytest to stdout (format json, default '|'); +ERROR: cannot specify DEFAULT in JSON mode +copy copytest to stdout (format json, header); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, header 1); +ERROR: cannot specify HEADER in JSON mode +copy copytest to stdout (format json, quote '"'); +ERROR: COPY QUOTE requires CSV mode +copy copytest to stdout (format json, escape '"'); +ERROR: COPY ESCAPE requires CSV mode +copy copytest to stdout (format json, force_quote *); +ERROR: COPY FORCE_QUOTE requires CSV mode +copy copytest to stdout (format json, force_not_null *); +ERROR: COPY FORCE_NOT_NULL requires CSV mode +copy copytest to stdout (format json, force_null *); +ERROR: COPY FORCE_NULL requires CSV mode +copy copytest to stdout (format json, on_error ignore); +ERROR: COPY ON_ERROR cannot be used with COPY TO +LINE 1: copy copytest to stdout (format json, on_error ignore); + ^ +copy copytest to stdout (format json, reject_limit 1); +ERROR: COPY REJECT_LIMIT requires ON_ERROR to be set to IGNORE +copy copytest from stdin(format json); +ERROR: COPY FORMAT JSON is not supported for COPY FROM +-- all of the above should yield error +-- column list with json format +copy copytest (style, test, filler) to stdout (format json); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); +-- full table +copy copyjsontest_types to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2},"arr":[1,2,3],"n":3.14,"b":true,"ts":"2024-01-15T10:30:00","t":"hello"} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}},"arr":[4,5],"n":-99.99,"b":false,"ts":"2024-06-30T23:59:59","t":"world"} +{"id":3,"js":null,"jsb":null,"arr":[],"n":null,"b":null,"ts":null,"t":null} +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +{"id":1,"js":{"a":1},"jsb":{"b": 2}} +{"id":2,"js":[1,null,"x"],"jsb":{"nested": {"k": "v"}}} +{"id":3,"js":null,"jsb":null} +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +{"id":1,"arr":[1,2,3],"n":3.14,"b":true} +{"id":2,"arr":[4,5],"n":-99.99,"b":false} +{"id":3,"arr":[],"n":null,"b":null} +copy copyjsontest_types (jsb, t) to stdout (format json); +{"jsb":{"b": 2},"t":"hello"} +{"jsb":{"nested": {"k": "v"}},"t":"world"} +{"jsb":null,"t":null} +copy copyjsontest_types (id, ts) to stdout (format json); +{"id":1,"ts":"2024-01-15T10:30:00"} +{"id":2,"ts":"2024-06-30T23:59:59"} +{"id":3,"ts":null} +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +{"js":{"a":1}} +{"js":[1,null,"x"]} +{"js":null} +copy copyjsontest_types (jsb) to stdout (format json); +{"jsb":{"b": 2}} +{"jsb":{"nested": {"k": "v"}}} +{"jsb":null} +drop table copyjsontest_types; +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); +copy copyjsontest to stdout json; +{"id":1,"f1":"line with \" in it: 1","f2":"1997-02-10T17:32:01-08:00"} +{"id":2,"f1":"line with ' in it: 2","f2":"1997-02-10T17:32:01-08:00"} +{"id":3,"f1":"line with \" in it: 3","f2":"1997-02-10T17:32:01-08:00"} +{"id":4,"f1":"line with ' in it: 4","f2":"1997-02-10T17:32:01-08:00"} +{"id":5,"f1":"line with \" in it: 5","f2":"1997-02-10T17:32:01-08:00"} +{"id":1,"f1":"aaa\"bbb","f2":null} +{"id":2,"f1":"aaa\\bbb","f2":null} +{"id":3,"f1":"aaa/bbb","f2":null} +{"id":4,"f1":"aaa\bbbb","f2":null} +{"id":5,"f1":"aaa\fbbb","f2":null} +{"id":6,"f1":"aaa\nbbb","f2":null} +{"id":7,"f1":"aaa\rbbb","f2":null} +{"id":8,"f1":"aaa\tbbb","f2":null} create temp table copytest4 ( c1 int, "colname with tab: " text); diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 65cbdaf7f3e..404f4321085 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -82,6 +82,94 @@ this is just a line full of junk that would error out if parsed copy copytest3 to stdout csv header; +--- test copying in JSON mode with various styles +copy (select 1 union all select 2) to stdout with (format json); +copy (select 1 as foo union all select 2) to stdout with (format json); +copy (values (1), (2)) TO stdout with (format json); +copy copytest to stdout json; +copy copytest to stdout (format json); +copy (select * from copytest) to stdout (format json); + +-- all of the following should yield error +copy copytest to stdout (format json, delimiter '|'); +copy copytest to stdout (format json, null '\N'); +copy copytest to stdout (format json, default '|'); +copy copytest to stdout (format json, header); +copy copytest to stdout (format json, header 1); +copy copytest to stdout (format json, quote '"'); +copy copytest to stdout (format json, escape '"'); +copy copytest to stdout (format json, force_quote *); +copy copytest to stdout (format json, force_not_null *); +copy copytest to stdout (format json, force_null *); +copy copytest to stdout (format json, on_error ignore); +copy copytest to stdout (format json, reject_limit 1); +copy copytest from stdin(format json); +-- all of the above should yield error + +-- column list with json format +copy copytest (style, test, filler) to stdout (format json); + +-- column list with diverse data types +create temp table copyjsontest_types ( + id int, + js json, + jsb jsonb, + arr int[], + n numeric(10,2), + b boolean, + ts timestamp, + t text); + +insert into copyjsontest_types values +(1, '{"a":1}', '{"b":2}', '{1,2,3}', 3.14, true, + '2024-01-15 10:30:00', 'hello'), +(2, '[1,null,"x"]', '{"nested":{"k":"v"}}', '{4,5}', -99.99, false, + '2024-06-30 23:59:59', 'world'), +(3, 'null', 'null', '{}', null, null, null, null); + +-- full table +copy copyjsontest_types to stdout (format json); + +-- column subsets exercising each type +copy copyjsontest_types (id, js, jsb) to stdout (format json); +copy copyjsontest_types (id, arr, n, b) to stdout (format json); +copy copyjsontest_types (jsb, t) to stdout (format json); +copy copyjsontest_types (id, ts) to stdout (format json); + +-- single column: json and jsonb +copy copyjsontest_types (js) to stdout (format json); +copy copyjsontest_types (jsb) to stdout (format json); + +drop table copyjsontest_types; + +-- embedded escaped characters +create temp table copyjsontest ( + id bigserial, + f1 text, + f2 timestamptz); + +insert into copyjsontest + select g.i, + CASE WHEN g.i % 2 = 0 THEN + 'line with '' in it: ' || g.i::text + ELSE + 'line with " in it: ' || g.i::text + END, + 'Mon Feb 10 17:32:01 1997 PST' + from generate_series(1,5) as g(i); + +insert into copyjsontest (f1) values +(E'aaa\"bbb'::text), +(E'aaa\\bbb'::text), +(E'aaa\/bbb'::text), +(E'aaa\bbbb'::text), +(E'aaa\fbbb'::text), +(E'aaa\nbbb'::text), +(E'aaa\rbbb'::text), +(E'aaa\tbbb'::text); + +copy copyjsontest to stdout json; + create temp table copytest4 ( c1 int, "colname with tab: " text); -- 2.43.0 [text/x-patch] v30-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (12.5K, 5-v30-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch) download | inline diff: From 9e0a65578eb1de9c8d0521b2931fbe76ad5f2bcf Mon Sep 17 00:00:00 2001 From: Andrew Dunstan <[email protected]> Date: Mon, 16 Mar 2026 16:51:12 -0400 Subject: [PATCH v30 3/3] Add option force_array for COPY JSON FORMAT This adds the force_array option, which is available exclusively when using COPY TO with the JSON format. When enabled, this option wraps the output in a top-level JSON array (enclosed in square brackets with comma-separated elements), making the entire result a valid single JSON value. Without this option, the default behavior is to output a stream of independent JSON objects. Attempting to use this option with COPY FROM or with formats other than JSON will raise an error. Author: Joe Conway <[email protected]> Author: jian he <[email protected]> Reviewed-by: Junwang Zhao <[email protected]> Reviewed-by: Masahiko Sawada <[email protected]> Reviewed-by: Florents Tselai <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> Discussion: https://postgr.es/m/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com Discussion: https://postgr.es/m/[email protected] --- doc/src/sgml/ref/copy.sgml | 30 +++++++++++++++++++++++ src/backend/commands/copy.c | 13 ++++++++++ src/backend/commands/copyto.c | 38 ++++++++++++++++++++++++++++-- src/bin/psql/tab-complete.in.c | 2 +- src/include/commands/copy.h | 1 + src/test/regress/expected/copy.out | 37 +++++++++++++++++++++++++++++ src/test/regress/sql/copy.sql | 13 ++++++++++ 7 files changed, 131 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml index 75f55bbf6f8..a79587f7613 100644 --- a/doc/src/sgml/ref/copy.sgml +++ b/doc/src/sgml/ref/copy.sgml @@ -40,6 +40,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable HEADER [ <replaceable class="parameter">boolean</replaceable> | <replaceable class="parameter">integer</replaceable> | MATCH ] QUOTE '<replaceable class="parameter">quote_character</replaceable>' ESCAPE '<replaceable class="parameter">escape_character</replaceable>' + FORCE_ARRAY [ <replaceable class="parameter">boolean</replaceable> ] FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * } @@ -366,6 +367,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable </listitem> </varlistentry> + <varlistentry id="sql-copy-params-force-array"> + <term><literal>FORCE_ARRAY</literal></term> + <listitem> + <para> + Force output of square brackets as array decorations at the beginning + and end of output, and commas between the rows. It is allowed only in + <command>COPY TO</command>, and only when using + <literal>json</literal> format. The default is + <literal>false</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-copy-params-force-quote"> <term><literal>FORCE_QUOTE</literal></term> <listitem> @@ -1103,6 +1117,22 @@ COPY country TO STDOUT (DELIMITER '|'); </programlisting> </para> +<para> + When the <literal>FORCE_ARRAY</literal> option is enabled, + the entire output is wrapped in a single JSON array with rows separated by commas: +<programlisting> +COPY (SELECT * FROM (VALUES(1),(2)) val(id)) TO STDOUT (FORMAT JSON, FORCE_ARRAY); +</programlisting> +The output is as follows: +<screen> +[ + {"id":1} +,{"id":2} +] +</screen> +</para> + + <para> To copy data from a file into the <literal>country</literal> table: <programlisting> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 29e22d91ecd..e837f417d0d 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -569,6 +569,7 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool force_array_specified = false; ListCell *option; /* Support external use for option sanity checking */ @@ -725,6 +726,13 @@ ProcessCopyOptions(ParseState *pstate, defel->defname), parser_errposition(pstate, defel->location))); } + else if (strcmp(defel->defname, "force_array") == 0) + { + if (force_array_specified) + errorConflictingDefElem(defel, pstate); + force_array_specified = true; + opts_out->force_array = defGetBoolean(defel); + } else if (strcmp(defel->defname, "on_error") == 0) { if (on_error_specified) @@ -967,6 +975,11 @@ ProcessCopyOptions(ParseState *pstate, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY %s is not supported for %s", "FORMAT JSON", "COPY FROM")); + if (opts_out->format != COPY_FORMAT_JSON && opts_out->force_array) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY %s can only be used with JSON mode", "FORCE_ARRAY")); + if (opts_out->default_print) { if (!is_from) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index ffe2268fbb0..12872f04eef 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -87,6 +87,7 @@ typedef struct CopyToStateData List *attnumlist; /* integer list of attnums to copy */ char *filename; /* filename, or NULL for STDOUT */ bool is_program; /* is 'filename' a program to popen? */ + bool json_row_delim_needed; /* need delimiter before next row */ StringInfo json_buf; /* reusable buffer for JSON output, * initialized in BeginCopyTo */ TupleDesc tupDesc; /* Descriptor for JSON output; for a column @@ -141,6 +142,7 @@ static void CopyToTextLikeOneRow(CopyToState cstate, TupleTableSlot *slot, bool is_csv); static void CopyToTextLikeEnd(CopyToState cstate); static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot); +static void CopyToJsonEnd(CopyToState cstate); static void CopyToBinaryStart(CopyToState cstate, TupleDesc tupDesc); static void CopyToBinaryOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo); static void CopyToBinaryOneRow(CopyToState cstate, TupleTableSlot *slot); @@ -182,7 +184,7 @@ static const CopyToRoutine CopyToRoutineJson = { .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToJsonOneRow, - .CopyToEnd = CopyToTextLikeEnd, + .CopyToEnd = CopyToJsonEnd, }; /* binary format */ @@ -248,6 +250,15 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) CopySendTextLikeEndOfRow(cstate); } + + /* + * If FORCE_ARRAY has been specified, send the opening bracket. + */ + if (cstate->opts.format == COPY_FORMAT_JSON && cstate->opts.force_array) + { + CopySendChar(cstate, '['); + CopySendTextLikeEndOfRow(cstate); + } } /* @@ -324,13 +335,24 @@ CopyToTextLikeOneRow(CopyToState cstate, CopySendTextLikeEndOfRow(cstate); } -/* Implementation of the end callback for text, CSV, and json formats */ +/* Implementation of the end callback for text and CSV formats */ static void CopyToTextLikeEnd(CopyToState cstate) { /* Nothing to do here */ } +/* Implementation of the end callback for json format */ +static void +CopyToJsonEnd(CopyToState cstate) +{ + if (cstate->opts.force_array) + { + CopySendChar(cstate, ']'); + CopySendTextLikeEndOfRow(cstate); + } +} + /* Implementation of per-row callback for json format */ static void CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) @@ -392,6 +414,18 @@ CopyToJsonOneRow(CopyToState cstate, TupleTableSlot *slot) composite_to_json(rowdata, cstate->json_buf, false); + if (cstate->opts.force_array) + { + if (cstate->json_row_delim_needed) + CopySendChar(cstate, ','); + else + { + /* first row needs no delimiter */ + CopySendChar(cstate, ' '); + cstate->json_row_delim_needed = true; + } + } + CopySendData(cstate, cstate->json_buf->data, cstate->json_buf->len); CopySendTextLikeEndOfRow(cstate); diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index ac36f4591f7..f44a1f22ebf 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -1232,7 +1232,7 @@ Copy_common_options, "DEFAULT", "FORCE_NOT_NULL", "FORCE_NULL", "FREEZE", \ /* COPY TO options */ #define Copy_to_options \ -Copy_common_options, "FORCE_QUOTE" +Copy_common_options, "FORCE_QUOTE", "FORCE_ARRAY" /* * These object types were introduced later than our support cutoff of diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 2b5bef6738e..abecfe51098 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -88,6 +88,7 @@ typedef struct CopyFormatOptions List *force_notnull; /* list of column names */ bool force_notnull_all; /* FORCE_NOT_NULL *? */ bool *force_notnull_flags; /* per-column CSV FNN flags */ + bool force_array; /* add JSON array decorations */ List *force_null; /* list of column names */ bool force_null_all; /* FORCE_NULL *? */ bool *force_null_flags; /* per-column CSV FN flags */ diff --git a/src/test/regress/expected/copy.out b/src/test/regress/expected/copy.out index 7f2d2e065f6..1714faab39c 100644 --- a/src/test/regress/expected/copy.out +++ b/src/test/regress/expected/copy.out @@ -83,6 +83,16 @@ copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); {"column1":1} {"column1":2} +copy (select 1 union all select 2) to stdout with (format json, force_array true); +[ + {"?column?":1} +,{"?column?":2} +] +copy (values (1), (2)) TO stdout with (format json, force_array true); +[ + {"column1":1} +,{"column1":2} +] copy copytest to stdout json; {"style":"DOS","test":"abc\r\ndef","filler":1} {"style":"Unix","test":"abc\ndef","filler":2} @@ -134,6 +144,33 @@ copy copytest (style, test, filler) to stdout (format json); {"style":"Unix","test":"abc\ndef","filler":2} {"style":"Mac","test":"abc\rdef","filler":3} {"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); +ERROR: COPY FORCE_ARRAY can only be used with JSON mode +-- force_array variants +copy copytest to stdout (format json, force_array); +[ + {"style":"DOS","test":"abc\r\ndef","filler":1} +,{"style":"Unix","test":"abc\ndef","filler":2} +,{"style":"Mac","test":"abc\rdef","filler":3} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +] +copy copytest(style, test) to stdout (format json, force_array true); +[ + {"style":"DOS","test":"abc\r\ndef"} +,{"style":"Unix","test":"abc\ndef"} +,{"style":"Mac","test":"abc\rdef"} +,{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb"} +] +copy copytest to stdout (format json, force_array false); +{"style":"DOS","test":"abc\r\ndef","filler":1} +{"style":"Unix","test":"abc\ndef","filler":2} +{"style":"Mac","test":"abc\rdef","filler":3} +{"style":"esc\\ape","test":"a\\r\\\r\\\n\\nb","filler":4} +-- force_array with empty result set +copy (select 1 where false) to stdout (format json, force_array); +[ +] -- column list with diverse data types create temp table copyjsontest_types ( id int, diff --git a/src/test/regress/sql/copy.sql b/src/test/regress/sql/copy.sql index 404f4321085..eaad290b257 100644 --- a/src/test/regress/sql/copy.sql +++ b/src/test/regress/sql/copy.sql @@ -86,6 +86,8 @@ copy copytest3 to stdout csv header; copy (select 1 union all select 2) to stdout with (format json); copy (select 1 as foo union all select 2) to stdout with (format json); copy (values (1), (2)) TO stdout with (format json); +copy (select 1 union all select 2) to stdout with (format json, force_array true); +copy (values (1), (2)) TO stdout with (format json, force_array true); copy copytest to stdout json; copy copytest to stdout (format json); copy (select * from copytest) to stdout (format json); @@ -109,6 +111,17 @@ copy copytest from stdin(format json); -- column list with json format copy copytest (style, test, filler) to stdout (format json); +-- should fail: force_array requires json format +copy copytest to stdout (format csv, force_array true); + +-- force_array variants +copy copytest to stdout (format json, force_array); +copy copytest(style, test) to stdout (format json, force_array true); +copy copytest to stdout (format json, force_array false); + +-- force_array with empty result set +copy (select 1 where false) to stdout (format json, force_array); + -- column list with diverse data types create temp table copyjsontest_types ( id int, -- 2.43.0 ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-17 17:50 Masahiko Sawada <[email protected]> parent: Andrew Dunstan <[email protected]> 1 sibling, 1 reply; 22+ messages in thread From: Masahiko Sawada @ 2026-03-17 17:50 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On Mon, Mar 16, 2026 at 1:59 PM Andrew Dunstan <[email protected]> wrote: > > > On 2026-03-16 Mo 2:24 PM, Masahiko Sawada wrote: > > On Sun, Mar 8, 2026 at 8:49 PM jian he <[email protected]> wrote: > > On Mon, Mar 9, 2026 at 3:44 AM Andrew Dunstan <[email protected]> wrote: > > Hmm. But should we be scribbling on slot->tts_tupleDescriptor like that? > How about something like this?: > > - * Full table or query without column list. Ensure the slot uses > - * cstate->tupDesc so that the datum is stamped with the right type; > - * for queries output type is RECORDOID this must be the blessed > - * descriptor so that composite_to_json can look it up via > - * lookup_rowtype_tupdesc. > + * Full table or query without column list. For queries, the slot's > + * TupleDesc may carry RECORDOID, which is not registered in the > type > + * cache and would cause composite_to_json's lookup_rowtype_tupdesc > + * call to fail. Build a HeapTuple stamped with the blessed > + * descriptor so the type can be looked up correctly. > */ > if (!cstate->rel && slot->tts_tupleDescriptor->tdtypeid == > RECORDOID) > - slot->tts_tupleDescriptor = cstate->queryDesc->tupDesc; > + { > + HeapTuple tup; > > - rowdata = ExecFetchSlotHeapTupleDatum(slot); > + tup = heap_form_tuple(cstate->tupDesc, > + slot->tts_values, > + slot->tts_isnull); > + rowdata = HeapTupleGetDatum(tup); > + } > + else > + { > + rowdata = ExecFetchSlotHeapTupleDatum(slot); > + } > > This is better. I've tried to get rid of json_projvalues and json_projnulls. > Just using heap_form_tuple, but it won't work. > > I incorporated the v28-0004 COPY column list into v9-0002. > With this patch set, we added four fields to the struct CopyToStateData. > > + StringInfo json_buf; /* reusable buffer for JSON output, > + * initialized in BeginCopyTo */ > + TupleDesc tupDesc; /* Descriptor for JSON output; for a column > + * list this is a projected descriptor */ > + Datum *json_projvalues; /* pre-allocated projection values, or > + * NULL */ > + bool *json_projnulls; /* pre-allocated projection nulls, or NULL */ > > Using the script in > https://www.postgresql.org/message-id/CACJufxFFZqxC3p4WjpTEi4riaJm%3DpADX%2Bpy0yQ0%3DRWTn5cqK3Q%40ma... > I tested it again on macOS and Linux, and there are no regressions for > COPY TO with the TEXT and CSV formats. > > I've reviewed the patch and have some comments: > > --- > I got a SEGV in the following scenario: > > postgres(1:1197708)=# create table test (a int, b text, c jsonb); > CREATE TABLE > postgres(1:1197708)=# copy test(a, b) to stdout with (format 'json' ); > TRAP: failed Assert("tupdesc->firstNonCachedOffsetAttr >= 0"), File: > "execTuples.c", Line: 2328, PID: 1197708 > postgres: masahiko postgres [local] COPY(ExceptionalCondition+0x9e) [0xbebe48] > postgres: masahiko postgres [local] COPY(BlessTupleDesc+0x2b) [0x729b50] > postgres: masahiko postgres [local] COPY(BeginCopyTo+0xc94) [0x637bdf] > postgres: masahiko postgres [local] COPY(DoCopy+0xb68) [0x62afbc] > postgres: masahiko postgres [local] > COPY(standard_ProcessUtility+0xa22) [0xa0ba48] > postgres: masahiko postgres [local] COPY(ProcessUtility+0x10e) [0xa0b01f] > postgres: masahiko postgres [local] COPY() [0xa09872] > postgres: masahiko postgres [local] COPY() [0xa09acf] > postgres: masahiko postgres [local] COPY(PortalRun+0x2c8) [0xa0901d] > postgres: masahiko postgres [local] COPY() [0xa02055] > postgres: masahiko postgres [local] COPY(PostgresMain+0xaf1) [0xa0724e] > postgres: masahiko postgres [local] COPY() [0x9fdab9] > postgres: masahiko postgres [local] > COPY(postmaster_child_launch+0x165) [0x905378] > postgres: masahiko postgres [local] COPY() [0x90b600] > postgres: masahiko postgres [local] COPY() [0x908e6a] > postgres: masahiko postgres [local] COPY(PostmasterMain+0x14fe) [0x90880c] > postgres: masahiko postgres [local] COPY(main+0x340) [0x7a1f9c] > > It seems to forget to call TupleDescFinalize(). And I think we need > some regression tests for this case. > > --- > + if (cstate->opts.format == COPY_FORMAT_JSON) > + { > + /* > + * If FORCE_ARRAY has been specified, send the opening bracket. > + */ > + if (cstate->opts.force_array) > + { > + CopySendChar(cstate, '['); > + CopySendTextLikeEndOfRow(cstate); > + } > + } > > We can conjunct the two if statement conditions. > > > Here's a v30 set that I hope fixes these issues. Thank you for updating the patch! The patches look good to me. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-17 19:49 Andrew Dunstan <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 0 replies; 22+ messages in thread From: Andrew Dunstan @ 2026-03-17 19:49 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: jian he <[email protected]>; Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Daniel Verite <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL-development <[email protected]> On 2026-03-17 Tu 1:50 PM, Masahiko Sawada wrote: >> >> Here's a v30 set that I hope fixes these issues. > Thank you for updating the patch! The patches look good to me. > Thanks. Planning to push this tomorrow. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com ^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Emitting JSON to file using COPY TO @ 2026-03-18 14:37 Daniel Verite <[email protected]> parent: Andrew Dunstan <[email protected]> 1 sibling, 0 replies; 22+ messages in thread From: Daniel Verite @ 2026-03-18 14:37 UTC (permalink / raw) To: Andrew Dunstan <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Jian He <[email protected]>; Joe Conway <[email protected]>; Junwang Zhao <[email protected]>; Florents Tselai <[email protected]>; Andrey M. Borodin <[email protected]>; Dean Rasheed <[email protected]>; Davin Shearer <[email protected]>; PostgreSQL development <[email protected]> Andrew Dunstan wrote: > Here's a v30 set that I hope fixes these issues. Currently there's no difference in output between the null json value and the SQL null. postgres=# create table tbl (j jsonb); CREATE TABLE postgres=# insert into tbl values('null'); INSERT 0 1 postgres=# insert into tbl values(null); INSERT 0 1 postgres=# table tbl; j ------ null (2 rows) postgres=# copy tbl to stdout with (format json); {"j":null} {"j":null} If we had to reload this file, we could not determine which kind of null we had even though they are different at the SQL level: postgres=# select null::jsonb is distinct from 'null'::jsonb; ?column? ---------- t Does it have to be that way or are there valid distinct outputs that we could use to avoid this ambiguity? Best regards, -- Daniel Vérité https://postgresql.verite.pro/ ^ permalink raw reply [nested|flat] 22+ messages in thread
end of thread, other threads:[~2026-03-18 14:37 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-11-10 00:53 Re: Emitting JSON to file using COPY TO jian he <[email protected]> 2025-11-29 02:46 ` jian he <[email protected]> 2026-01-21 12:15 ` jian he <[email protected]> 2026-02-03 16:40 ` Florents Tselai <[email protected]> 2026-02-06 03:26 ` jian he <[email protected]> 2026-02-06 06:19 ` Junwang Zhao <[email protected]> 2026-02-07 13:27 ` Junwang Zhao <[email protected]> 2026-02-09 03:48 ` jian he <[email protected]> 2026-03-04 15:51 ` Andrew Dunstan <[email protected]> 2026-03-05 18:06 ` Joe Conway <[email protected]> 2026-03-05 20:49 ` Andrew Dunstan <[email protected]> 2026-03-06 09:38 ` jian he <[email protected]> 2026-03-06 12:33 ` Andrew Dunstan <[email protected]> 2026-03-08 16:16 ` jian he <[email protected]> 2026-03-08 19:44 ` Andrew Dunstan <[email protected]> 2026-03-09 03:48 ` jian he <[email protected]> 2026-03-09 12:23 ` Andrew Dunstan <[email protected]> 2026-03-16 18:24 ` Masahiko Sawada <[email protected]> 2026-03-16 20:59 ` Andrew Dunstan <[email protected]> 2026-03-17 17:50 ` Masahiko Sawada <[email protected]> 2026-03-17 19:49 ` Andrew Dunstan <[email protected]> 2026-03-18 14:37 ` Daniel Verite <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox