public inbox for [email protected]
help / color / mirror / Atom feedFrom: Joel Jacobson <[email protected]>
To: [email protected]
Subject: New COPY options: DELIMITER NONE and QUOTE NONE
Date: Fri, 19 May 2023 11:24:27 +0200
Message-ID: <[email protected]> (raw)
The thread "Should CSV parsing be stricter about mid-field quotes?" [1] forked
into a new topic, with two new ideas, hence this new thread.
1. COPY ... QUOTE NONE
In the [1] thread, Andrew Dunstan suggested a trick on how to deal with
unquoted but delimited files, such as TSV-files produced by Google Sheets:
> You can use CSV mode pretty reliably for TSV files.
> The trick is to use a quoting char that shouldn't appear,
> such as E'\x01' as well as setting the delimiter to E'\t'.
> Yes, it's far from obvious.
Would it be an improvement to allow specifying `QUOTE NONE` instead?
quotes.tsv:
id quote
1 "E = mc^2" -- Albert Einstein
COPY quotes FROM '/tmp/quotes.tsv' WITH CSV HEADER DELIMITER E'\t' QUOTE NONE;
SELECT * FROM quotes;
id | quote
----+-------------------------------
1 | "E = mc^2" -- Albert Einstein
(1 row)
2. COPY ... DELIMITER NONE
This is meant to improve the use-case when wanting to import e.g. an
unstructured log file line-by-line into a single column.
The current trick I've been using is similar to the first one,
that is, to specify a non-existing delimiter. But that involves having to find
some non-existing byte, which is error-prone since future log files might
suddenly start to contain it. So I think it would be better to be to be explicit
about not wanting to delimit fields at all, treating the entire whole line as a column.
Example:
% cat /tmp/apache.log
192.168.1.1 - - [19/May/2023:09:54:17 -0700] "GET /index.html HTTP/1.1" 200 431 "http://www.example.com/home.html"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
192.168.1.2 - - [19/May/2023:09:55:12 -0700] "POST /form.php HTTP/1.1" 200 512 "http://www.example.com/form.html"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
CREATE TABLE unstructured_log (whole_line text);
COPY unstructured_log FROM '/tmp/apache.log' WITH CSV DELIMITER NONE QUOTE NONE;
SELECT * FROM unstructured_log;
whole_line
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
192.168.1.1 - - [19/May/2023:09:54:17 -0700] "GET /index.html HTTP/1.1" 200 431 "http://www.example.com/home.html"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
192.168.1.2 - - [19/May/2023:09:55:12 -0700] "POST /form.php HTTP/1.1" 200 512 "http://www.example.com/form.html"; "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
(2 rows)
I hacked together a broken patch just to demonstrate the idea on syntax
and basic idea. The `COPY ... FROM` examples above works.
But it doesn't work at all for `COPY ... TO`, since it output \0 byte as
delimiter and quote in the output, which is of course not what we want.
Just wanted some feedback to see if there is any broader interest in this,
before proceeding and looking into how to implement it properly.
Is this something we want or are there just a few of us who have needed this in the past?
/Joel
[1] https://www.postgresql.org/message-id/31c81233-d707-0d2a-8111-a915f463459b%40dunslane.net
Attachments:
[application/octet-stream] copy_delimiter_quote_none.patch (3.3K, ../[email protected]/3-copy_delimiter_quote_none.patch)
download | inline diff:
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f14fae3308..f4348a6197 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -611,7 +611,7 @@ ProcessCopyOptions(ParseState *pstate,
}
/* Only single-byte delimiter strings are supported. */
- if (strlen(opts_out->delim) != 1)
+ if (strlen(opts_out->delim) > 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single one-byte character")));
@@ -669,12 +669,15 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote available only in CSV mode")));
- if (opts_out->csv_mode && strlen(opts_out->quote) != 1)
+ if (opts_out->csv_mode && strlen(opts_out->quote) > 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote must be a single one-byte character")));
- if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0])
+ if (opts_out->csv_mode &&
+ opts_out->delim[0] == opts_out->quote[0] &&
+ opts_out->delim[0] != '\0' &&
+ opts_out->quote[0] != '\0')
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("COPY delimiter and quote must be different")));
@@ -685,7 +688,7 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape available only in CSV mode")));
- if (opts_out->csv_mode && strlen(opts_out->escape) != 1)
+ if (opts_out->csv_mode && strlen(opts_out->escape) > 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape must be a single one-byte character")));
@@ -722,14 +725,16 @@ ProcessCopyOptions(ParseState *pstate,
errmsg("COPY force null only available using COPY FROM")));
/* Don't allow the delimiter to appear in the null string. */
- if (strchr(opts_out->null_print, opts_out->delim[0]) != NULL)
+ if (strchr(opts_out->null_print, opts_out->delim[0]) != NULL &&
+ opts_out->delim[0] != '\0')
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must not appear in the NULL specification")));
/* Don't allow the CSV quote char to appear in the null string. */
if (opts_out->csv_mode &&
- strchr(opts_out->null_print, opts_out->quote[0]) != NULL)
+ strchr(opts_out->null_print, opts_out->quote[0]) != NULL &&
+ opts_out->quote[0] != '\0')
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("CSV quote character must not appear in the NULL specification")));
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6f5aa8a3cb..e0c2002b50 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3409,6 +3409,10 @@ copy_opt_item:
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
}
+ | DELIMITER NONE
+ {
+ $$ = makeDefElem("delimiter", (Node *) makeString("\0"), @1);
+ }
| NULL_P opt_as Sconst
{
$$ = makeDefElem("null", (Node *) makeString($3), @1);
@@ -3425,6 +3429,10 @@ copy_opt_item:
{
$$ = makeDefElem("quote", (Node *) makeString($3), @1);
}
+ | QUOTE NONE
+ {
+ $$ = makeDefElem("quote", (Node *) makeString("\0"), @1);
+ }
| ESCAPE opt_as Sconst
{
$$ = makeDefElem("escape", (Node *) makeString($3), @1);
view thread (5+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: New COPY options: DELIMITER NONE and QUOTE NONE
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox