public inbox for [email protected]
help / color / mirror / Atom feedPOC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
117+ messages / 21 participants
[nested] [flat]
* POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
@ 2014-12-25 21:23 Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alex Shulgin @ 2014-12-25 21:23 UTC (permalink / raw)
To: pgsql-hackers
Trent Shipley <[email protected]> writes:
> On Friday 2007-12-14 16:22, Tom Lane wrote:
>> Neil Conway <[email protected]> writes:
>> > By modifying COPY: COPY IGNORE ERRORS or some such would instruct COPY
>> > to drop (and log) rows that contain malformed data. That is, rows with
>> > too many or too few columns, rows that result in constraint violations,
>> > and rows containing columns where the data type's input function raises
>> > an error. The last case is the only thing that would be a bit tricky to
>> > implement, I think: you could use PG_TRY() around the InputFunctionCall,
>> > but I guess you'd need a subtransaction to ensure that you reset your
>> > state correctly after catching an error.
>>
>> Yeah. It's the subtransaction per row that's daunting --- not only the
>> cycles spent for that, but the ensuing limitation to 4G rows imported
>> per COPY.
>
> You could extend the COPY FROM syntax with a COMMIT EVERY n clause. This
> would help with the 4G subtransaction limit. The cost to the ETL process is
> that a simple rollback would not be guaranteed send the process back to it's
> initial state. There are easy ways to deal with the rollback issue though.
>
> A {NO} RETRY {USING algorithm} clause might be useful. If the NO RETRY
> option is selected then the COPY FROM can run without subtransactions and in
> excess of the 4G per transaction limit. NO RETRY should be the default since
> it preserves the legacy behavior of COPY FROM.
>
> You could have an EXCEPTIONS TO {filename|STDERR} clause. I would not give the
> option of sending exceptions to a table since they are presumably malformed,
> otherwise they would not be exceptions. (Users should re-process exception
> files if they want an if good then table a else exception to table b ...)
>
> EXCEPTIONS TO and NO RETRY would be mutually exclusive.
>
>
>> If we could somehow only do a subtransaction per failure, things would
>> be much better, but I don't see how.
Hello,
Attached is a proof of concept patch for this TODO item. There is no
docs yet, I just wanted to know if approach is sane.
The added syntax is like the following:
COPY [table] FROM [file/program/stdin] EXCEPTIONS TO [file or stdout]
The way it's done it is abusing Copy Both mode and from my limited
testing, that seems to just work. The error trapping itself is done
using PG_TRY/PG_CATCH and can only catch formatting or before-insert
trigger errors, no attempt is made to recover from a failed unique
constraint, etc.
Example in action:
postgres=# \d test_copy2
Table "public.test_copy2"
Column | Type | Modifiers
--------+---------+-----------
id | integer |
val | integer |
postgres=# copy test_copy2 from program 'seq 3' exceptions to stdout;
1
NOTICE: missing data for column "val"
CONTEXT: COPY test_copy2, line 1: "1"
2
NOTICE: missing data for column "val"
CONTEXT: COPY test_copy2, line 2: "2"
3
NOTICE: missing data for column "val"
CONTEXT: COPY test_copy2, line 3: "3"
NOTICE: total exceptions ignored: 3
postgres=# \d test_copy1
Table "public.test_copy1"
Column | Type | Modifiers
--------+---------+-----------
id | integer | not null
postgres=# set client_min_messages to warning;
SET
postgres=# copy test_copy1 from program 'ls /proc' exceptions to stdout;
...
vmstat
zoneinfo
postgres=#
Limited performance testing shows no significant difference between
error-catching and plain code path. For example, timing
copy test_copy1 from program 'seq 1000000' [exceptions to stdout]
shows similar numbers with or without the added "exceptions to" clause.
Now that I'm sending this I wonder if the original comment about the
need for subtransaction around every loaded line still holds. Any
example of what would be not properly rolled back by just PG_TRY?
Happy hacking!
--
Alex
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[text/x-diff] 0001-poc-copy-from-.-exceptions-to.patch (37.7K, ../../[email protected]/2-0001-poc-copy-from-.-exceptions-to.patch)
download | inline diff:
From 50f7ab0a503a0d61776add8a138abf2622fc6c35 Mon Sep 17 00:00:00 2001
From: Alex Shulgin <[email protected]>
Date: Fri, 19 Dec 2014 18:21:31 +0300
Subject: [PATCH] POC: COPY FROM ... EXCEPTIONS TO
---
contrib/file_fdw/file_fdw.c | 4 +-
src/backend/commands/copy.c | 251 +++++++++++++++++++++++++++++---
src/backend/parser/gram.y | 26 +++-
src/bin/psql/common.c | 14 +-
src/bin/psql/copy.c | 119 ++++++++++++++-
src/bin/psql/settings.h | 1 +
src/bin/psql/startup.c | 1 +
src/bin/psql/tab-complete.c | 12 +-
src/include/commands/copy.h | 3 +-
src/include/nodes/parsenodes.h | 1 +
src/include/parser/kwlist.h | 1 +
src/interfaces/ecpg/preproc/ecpg.addons | 2 +-
12 files changed, 396 insertions(+), 39 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
new file mode 100644
index 5a4d5aa..0df02f7
*** a/contrib/file_fdw/file_fdw.c
--- b/contrib/file_fdw/file_fdw.c
*************** fileBeginForeignScan(ForeignScanState *n
*** 624,629 ****
--- 624,630 ----
cstate = BeginCopyFrom(node->ss.ss_currentRelation,
filename,
false,
+ NULL,
NIL,
options);
*************** fileReScanForeignScan(ForeignScanState *
*** 697,702 ****
--- 698,704 ----
festate->cstate = BeginCopyFrom(node->ss.ss_currentRelation,
festate->filename,
false,
+ NULL,
NIL,
festate->options);
}
*************** file_acquire_sample_rows(Relation onerel
*** 1030,1036 ****
/*
* Create CopyState from FDW options.
*/
! cstate = BeginCopyFrom(onerel, filename, false, NIL, options);
/*
* Use per-tuple memory context to prevent leak of memory used to read
--- 1032,1038 ----
/*
* Create CopyState from FDW options.
*/
! cstate = BeginCopyFrom(onerel, filename, false, NULL, NIL, options);
/*
* Use per-tuple memory context to prevent leak of memory used to read
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
new file mode 100644
index 08abe14..4f59c63
*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
*************** typedef enum EolType
*** 96,102 ****
typedef struct CopyStateData
{
/* low-level state data */
! CopyDest copy_dest; /* type of copy source/destination */
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
--- 96,103 ----
typedef struct CopyStateData
{
/* low-level state data */
! CopyDest copy_src; /* type of copy source */
! CopyDest copy_dest; /* type of copy destination */
FILE *copy_file; /* used if copy_dest == COPY_FILE */
StringInfo fe_msgbuf; /* used for all dests during COPY TO, only for
* dest == COPY_NEW_FE in COPY FROM */
*************** typedef struct CopyStateData
*** 105,110 ****
--- 106,114 ----
int file_encoding; /* file or remote side's character encoding */
bool need_transcoding; /* file encoding diff from server? */
bool encoding_embeds_ascii; /* ASCII can be non-first byte? */
+ bool ignore_exceptions; /* should we trap and ignore exceptions? */
+ FILE *exc_file; /* file stream to write erroring lines to */
+ uint64 exceptions; /* total number of exceptions ignored */
/* parameters from the COPY command */
Relation rel; /* relation to copy to or from */
*************** typedef struct CopyStateData
*** 112,117 ****
--- 116,122 ----
List *attnumlist; /* integer list of attnums to copy */
char *filename; /* filename, or NULL for STDIN/STDOUT */
bool is_program; /* is 'filename' a program to popen? */
+ char *exc_filename; /* filename for exceptions or NULL for STDOUT */
bool binary; /* binary format? */
bool oids; /* include OIDs? */
bool freeze; /* freeze rows on loading? */
*************** SendCopyBegin(CopyState cstate)
*** 347,352 ****
--- 352,366 ----
int16 format = (cstate->binary ? 1 : 0);
int i;
+ /*
+ * Check if we might need to stream exceptions to the frontend. If
+ * so, this must be a "COPY FROM file/program EXCEPTIONS TO STDOUT".
+ *
+ * We need to create the frontend message buffer now.
+ */
+ if (cstate->ignore_exceptions)
+ cstate->fe_msgbuf = makeStringInfo();
+
pq_beginmessage(&buf, 'H');
pq_sendbyte(&buf, format); /* overall format */
pq_sendint(&buf, natts, 2);
*************** ReceiveCopyBegin(CopyState cstate)
*** 388,404 ****
{
/* new way */
StringInfoData buf;
int natts = list_length(cstate->attnumlist);
int16 format = (cstate->binary ? 1 : 0);
int i;
! pq_beginmessage(&buf, 'G');
pq_sendbyte(&buf, format); /* overall format */
pq_sendint(&buf, natts, 2);
for (i = 0; i < natts; i++)
pq_sendint(&buf, format, 2); /* per-column formats */
pq_endmessage(&buf);
! cstate->copy_dest = COPY_NEW_FE;
cstate->fe_msgbuf = makeStringInfo();
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
--- 402,428 ----
{
/* new way */
StringInfoData buf;
+ char msgid = 'G'; /* receiving from client only */
int natts = list_length(cstate->attnumlist);
int16 format = (cstate->binary ? 1 : 0);
int i;
! /*
! * Check if we also need to pipe exceptions back to the frontend.
! */
! if (cstate->ignore_exceptions && cstate->exc_filename == NULL)
! {
! msgid = 'W'; /* copying in both directions */
! cstate->copy_dest = COPY_NEW_FE;
! }
!
! pq_beginmessage(&buf, msgid);
pq_sendbyte(&buf, format); /* overall format */
pq_sendint(&buf, natts, 2);
for (i = 0; i < natts; i++)
pq_sendint(&buf, format, 2); /* per-column formats */
pq_endmessage(&buf);
! cstate->copy_src = COPY_NEW_FE;
cstate->fe_msgbuf = makeStringInfo();
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
*************** ReceiveCopyBegin(CopyState cstate)
*** 409,415 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
! cstate->copy_dest = COPY_OLD_FE;
}
else
{
--- 433,439 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
! cstate->copy_src = COPY_OLD_FE;
}
else
{
*************** ReceiveCopyBegin(CopyState cstate)
*** 419,425 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
! cstate->copy_dest = COPY_OLD_FE;
}
/* We *must* flush here to ensure FE knows it can send. */
pq_flush();
--- 443,449 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
! cstate->copy_src = COPY_OLD_FE;
}
/* We *must* flush here to ensure FE knows it can send. */
pq_flush();
*************** CopySendChar(CopyState cstate, char c)
*** 472,486 ****
appendStringInfoCharMacro(cstate->fe_msgbuf, c);
}
static void
CopySendEndOfRow(CopyState cstate)
{
StringInfo fe_msgbuf = cstate->fe_msgbuf;
switch (cstate->copy_dest)
{
case COPY_FILE:
! if (!cstate->binary)
{
/* Default line termination depends on platform */
#ifndef WIN32
--- 496,560 ----
appendStringInfoCharMacro(cstate->fe_msgbuf, c);
}
+ /*
+ * This should be called from PG_CATCH() after switching to appropriate
+ * MemoryContext.
+ */
+ static void
+ CopySendException(CopyState cstate)
+ {
+ ErrorData *error;
+
+ ++cstate->exceptions;
+
+ /*
+ * When reading from the frontend, we reuse the current line held in the
+ * message buffer to send the exception line back, otherwise we need to
+ * copy the line over from the line buffer.
+ */
+ if (cstate->copy_src == COPY_FILE)
+ CopySendData(cstate, cstate->line_buf.data, cstate->line_buf.len);
+
+ /* this flushes the message buffer */
+ CopySendEndOfRow(cstate);
+
+ error = CopyErrorData();
+ FlushErrorState();
+
+ /* report error as a harmless notice */
+ ereport(NOTICE,
+ (errmsg("%s", error->message)));
+ FreeErrorData(error);
+ }
+
static void
CopySendEndOfRow(CopyState cstate)
{
StringInfo fe_msgbuf = cstate->fe_msgbuf;
+ FILE *file;
+ bool should_add_newline;
+
+ /* determine where are we writing to */
+ if (cstate->ignore_exceptions)
+ {
+ file = cstate->exc_file;
+ /*
+ * We should only add a newline if we're not sending the frontend what
+ * it has just sent us and in any case we shouldn't do this for binary
+ * copy.
+ */
+ should_add_newline = (cstate->copy_src == COPY_FILE && !cstate->binary);
+ }
+ else
+ {
+ file = cstate->copy_file;
+ should_add_newline = !(cstate->binary);
+ }
switch (cstate->copy_dest)
{
case COPY_FILE:
! if (should_add_newline)
{
/* Default line termination depends on platform */
#ifndef WIN32
*************** CopySendEndOfRow(CopyState cstate)
*** 490,498 ****
#endif
}
! if (fwrite(fe_msgbuf->data, fe_msgbuf->len, 1,
! cstate->copy_file) != 1 ||
! ferror(cstate->copy_file))
{
if (cstate->is_program)
{
--- 564,571 ----
#endif
}
! if (fwrite(fe_msgbuf->data, fe_msgbuf->len, 1, file) != 1 ||
! ferror(file))
{
if (cstate->is_program)
{
*************** CopySendEndOfRow(CopyState cstate)
*** 525,531 ****
break;
case COPY_OLD_FE:
/* The FE/BE protocol uses \n as newline for all platforms */
! if (!cstate->binary)
CopySendChar(cstate, '\n');
if (pq_putbytes(fe_msgbuf->data, fe_msgbuf->len))
--- 598,604 ----
break;
case COPY_OLD_FE:
/* The FE/BE protocol uses \n as newline for all platforms */
! if (should_add_newline)
CopySendChar(cstate, '\n');
if (pq_putbytes(fe_msgbuf->data, fe_msgbuf->len))
*************** CopySendEndOfRow(CopyState cstate)
*** 538,544 ****
break;
case COPY_NEW_FE:
/* The FE/BE protocol uses \n as newline for all platforms */
! if (!cstate->binary)
CopySendChar(cstate, '\n');
/* Dump the accumulated row as one CopyData message */
--- 611,617 ----
break;
case COPY_NEW_FE:
/* The FE/BE protocol uses \n as newline for all platforms */
! if (should_add_newline)
CopySendChar(cstate, '\n');
/* Dump the accumulated row as one CopyData message */
*************** CopySendEndOfRow(CopyState cstate)
*** 546,552 ****
break;
}
! resetStringInfo(fe_msgbuf);
}
/*
--- 619,630 ----
break;
}
! /*
! * Avoid resetting the buffer we reused to send the exception line back to
! * the frontend.
! */
! if (!cstate->ignore_exceptions || cstate->copy_src == COPY_FILE)
! resetStringInfo(fe_msgbuf);
}
/*
*************** CopyGetData(CopyState cstate, void *data
*** 567,573 ****
{
int bytesread = 0;
! switch (cstate->copy_dest)
{
case COPY_FILE:
bytesread = fread(databuf, 1, maxread, cstate->copy_file);
--- 645,651 ----
{
int bytesread = 0;
! switch (cstate->copy_src)
{
case COPY_FILE:
bytesread = fread(databuf, 1, maxread, cstate->copy_file);
*************** DoCopy(const CopyStmt *stmt, const char
*** 919,930 ****
PreventCommandIfReadOnly("COPY FROM");
cstate = BeginCopyFrom(rel, stmt->filename, stmt->is_program,
! stmt->attlist, stmt->options);
*processed = CopyFrom(cstate); /* copy from file to database */
EndCopyFrom(cstate);
}
else
{
cstate = BeginCopyTo(rel, query, queryString, relid,
stmt->filename, stmt->is_program,
stmt->attlist, stmt->options);
--- 997,1018 ----
PreventCommandIfReadOnly("COPY FROM");
cstate = BeginCopyFrom(rel, stmt->filename, stmt->is_program,
! stmt->exc_filename, stmt->attlist, stmt->options);
*processed = CopyFrom(cstate); /* copy from file to database */
+ if (cstate->exceptions)
+ ereport(NOTICE,
+ (errmsg("total exceptions ignored: " UINT64_FORMAT,
+ cstate->exceptions)));
EndCopyFrom(cstate);
}
else
{
+ if (stmt->exc_filename != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("EXCEPTIONS TO not allowed with COPY ... TO"),
+ errhint("see COPY ... FROM")));
+
cstate = BeginCopyTo(rel, query, queryString, relid,
stmt->filename, stmt->is_program,
stmt->attlist, stmt->options);
*************** BeginCopy(bool is_from,
*** 1561,1566 ****
--- 1649,1655 ----
/* See Multibyte encoding comment above */
cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->file_encoding);
+ cstate->copy_src = COPY_FILE; /* default */
cstate->copy_dest = COPY_FILE; /* default */
MemoryContextSwitchTo(oldcontext);
*************** EndCopy(CopyState cstate)
*** 1608,1613 ****
--- 1697,1708 ----
cstate->filename)));
}
+ if (cstate->exc_filename != NULL && FreeFile(cstate->exc_file))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m",
+ cstate->exc_filename)));
+
MemoryContextDelete(cstate->copycontext);
pfree(cstate);
}
*************** CopyFrom(CopyState cstate)
*** 2331,2336 ****
--- 2426,2432 ----
{
TupleTableSlot *slot;
bool skip_tuple;
+ bool depleted;
Oid loaded_oid = InvalidOid;
CHECK_FOR_INTERRUPTS();
*************** CopyFrom(CopyState cstate)
*** 2348,2356 ****
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! if (!NextCopyFrom(cstate, econtext, values, nulls, &loaded_oid))
break;
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
--- 2444,2475 ----
/* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! skip_tuple = false;
! depleted = false;
!
! PG_TRY();
! {
! if (!NextCopyFrom(cstate, econtext, values, nulls, &loaded_oid))
! /* can't break right here due to PG_TRY using do/while(0) */
! depleted = true;
! }
! PG_CATCH();
! {
! if (!cstate->ignore_exceptions)
! PG_RE_THROW();
!
! skip_tuple = true;
! MemoryContextSwitchTo(oldcontext);
! CopySendException(cstate);
! }
! PG_END_TRY();
!
! if (depleted)
break;
+ if (skip_tuple)
+ continue;
+
/* And now we can form the input tuple. */
tuple = heap_form_tuple(tupDesc, values, nulls);
*************** CopyFrom(CopyState cstate)
*** 2376,2382 ****
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->trig_insert_before_row)
{
! slot = ExecBRInsertTriggers(estate, resultRelInfo, slot);
if (slot == NULL) /* "do nothing" */
skip_tuple = true;
--- 2495,2514 ----
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->trig_insert_before_row)
{
! PG_TRY();
! {
! slot = ExecBRInsertTriggers(estate, resultRelInfo, slot);
! }
! PG_CATCH();
! {
! if (!cstate->ignore_exceptions)
! PG_RE_THROW();
!
! slot = NULL;
! MemoryContextSwitchTo(oldcontext);
! CopySendException(cstate);
! }
! PG_END_TRY();
if (slot == NULL) /* "do nothing" */
skip_tuple = true;
*************** CopyFrom(CopyState cstate)
*** 2384,2395 ****
tuple = ExecMaterializeSlot(slot);
}
! if (!skip_tuple)
{
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
if (useHeapMultiInsert)
{
/* Add this tuple to the tuple buffer */
--- 2516,2545 ----
tuple = ExecMaterializeSlot(slot);
}
! if (skip_tuple)
! continue;
!
! /* Check the constraints of the tuple */
! if (cstate->rel->rd_att->constr)
{
! PG_TRY();
! {
ExecConstraints(resultRelInfo, slot, estate);
+ }
+ PG_CATCH();
+ {
+ if (!cstate->ignore_exceptions)
+ PG_RE_THROW();
+ skip_tuple = true;
+ MemoryContextSwitchTo(oldcontext);
+ CopySendException(cstate);
+ }
+ PG_END_TRY();
+ }
+
+ if (!skip_tuple)
+ {
if (useHeapMultiInsert)
{
/* Add this tuple to the tuple buffer */
*************** CopyState
*** 2573,2583 ****
--- 2723,2736 ----
BeginCopyFrom(Relation rel,
const char *filename,
bool is_program,
+ const char *exc_filename,
List *attnamelist,
List *options)
{
CopyState cstate;
bool pipe = (filename == NULL);
+ bool ignore_exceptions = (exc_filename != NULL);
+ bool exc_pipe = (exc_filename != NULL && *exc_filename == 0);
TupleDesc tupDesc;
Form_pg_attribute *attr;
AttrNumber num_phys_attrs,
*************** BeginCopyFrom(Relation rel,
*** 2686,2691 ****
--- 2839,2900 ----
cstate->volatile_defexprs = volatile_defexprs;
cstate->num_defaults = num_defaults;
cstate->is_program = is_program;
+ cstate->ignore_exceptions = ignore_exceptions;
+
+ if (ignore_exceptions)
+ {
+ if (exc_pipe)
+ {
+ if (whereToSendOutput == DestRemote)
+ {
+ if (!pipe)
+ SendCopyBegin(cstate);
+ else
+ ; /* handled by ReceiveCopyBegin() call below */
+ }
+ else
+ {
+ cstate->exc_file = stdout;
+ }
+ }
+ else
+ {
+ if (!is_absolute_path(exc_filename))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("relative path not allowed for EXCEPTIONS TO file")));
+
+ if (!pipe)
+ {
+ struct stat stbuf_exc;
+ struct stat stbuf_from;
+
+ /* check if both FROM and EXCEPTIONS TO are the same file */
+ if (stat(exc_filename, &stbuf_exc) == 0 &&
+ stat(filename, &stbuf_from) == 0 &&
+ stbuf_exc.st_dev == stbuf_from.st_dev &&
+ stbuf_exc.st_ino == stbuf_from.st_ino)
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_NAME),
+ errmsg("cannot specify the same file for COPY FROM and EXCEPTIONS TO")));
+ }
+ /*
+ * We won't send or receive data via frontend, but we still
+ * need to the buffer for CopySendException() to work with.
+ */
+ cstate->fe_msgbuf = makeStringInfo();
+ }
+
+ cstate->exc_filename = pstrdup(exc_filename);
+ cstate->exc_file = AllocateFile(cstate->exc_filename, PG_BINARY_W);
+ if (cstate->exc_file == NULL)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\" for writing: %m",
+ cstate->exc_filename)));
+ }
+ }
if (pipe)
{
*************** NextCopyFrom(CopyState cstate, ExprConte
*** 3019,3025 ****
*/
char dummy;
! if (cstate->copy_dest != COPY_OLD_FE &&
CopyGetData(cstate, &dummy, 1, 1) > 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 3228,3234 ----
*/
char dummy;
! if (cstate->copy_src != COPY_OLD_FE &&
CopyGetData(cstate, &dummy, 1, 1) > 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
*************** CopyReadLine(CopyState cstate)
*** 3133,3139 ****
* after \. up to the protocol end of copy data. (XXX maybe better
* not to treat \. as special?)
*/
! if (cstate->copy_dest == COPY_NEW_FE)
{
do
{
--- 3342,3348 ----
* after \. up to the protocol end of copy data. (XXX maybe better
* not to treat \. as special?)
*/
! if (cstate->copy_src == COPY_NEW_FE)
{
do
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
new file mode 100644
index 6431601..47f2be2
*** a/src/backend/parser/gram.y
--- b/src/backend/parser/gram.y
*************** static Node *makeRecursiveViewSelect(cha
*** 307,313 ****
%type <defelt> event_trigger_when_item
%type <chr> enable_trigger
! %type <str> copy_file_name
database_name access_method_clause access_method attr_name
name cursor_name file_name
index_name opt_index_name cluster_index_specification
--- 307,313 ----
%type <defelt> event_trigger_when_item
%type <chr> enable_trigger
! %type <str> copy_file_name opt_copy_exceptions
database_name access_method_clause access_method attr_name
name cursor_name file_name
index_name opt_index_name cluster_index_specification
*************** static Node *makeRecursiveViewSelect(cha
*** 561,567 ****
DEFERRABLE DEFERRED DEFINER DELETE_P DELIMITER DELIMITERS DESC
DICTIONARY DISABLE_P DISCARD DISTINCT DO DOCUMENT_P DOMAIN_P DOUBLE_P DROP
! EACH ELSE ENABLE_P ENCODING ENCRYPTED END_P ENUM_P ESCAPE EVENT EXCEPT
EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN
EXTENSION EXTERNAL EXTRACT
--- 561,568 ----
DEFERRABLE DEFERRED DEFINER DELETE_P DELIMITER DELIMITERS DESC
DICTIONARY DISABLE_P DISCARD DISTINCT DO DOCUMENT_P DOMAIN_P DOUBLE_P DROP
! EACH ELSE ENABLE_P ENCODING ENCRYPTED END_P ENUM_P ESCAPE EVENT
! EXCEPT EXCEPTIONS
EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN
EXTENSION EXTERNAL EXTRACT
*************** ClosePortalStmt:
*** 2515,2521 ****
/*****************************************************************************
*
* QUERY :
! * COPY relname [(columnList)] FROM/TO file [WITH] [(options)]
* COPY ( SELECT ... ) TO file [WITH] [(options)]
*
* where 'file' can be one of:
--- 2516,2523 ----
/*****************************************************************************
*
* QUERY :
! * COPY relname [(columnList)] FROM/TO file [EXCEPTIONS TO file]
! * [WITH] [(options)]
* COPY ( SELECT ... ) TO file [WITH] [(options)]
*
* where 'file' can be one of:
*************** ClosePortalStmt:
*** 2534,2540 ****
*****************************************************************************/
CopyStmt: COPY opt_binary qualified_name opt_column_list opt_oids
! copy_from opt_program copy_file_name copy_delimiter opt_with copy_options
{
CopyStmt *n = makeNode(CopyStmt);
n->relation = $3;
--- 2536,2543 ----
*****************************************************************************/
CopyStmt: COPY opt_binary qualified_name opt_column_list opt_oids
! copy_from opt_program copy_file_name opt_copy_exceptions copy_delimiter
! opt_with copy_options
{
CopyStmt *n = makeNode(CopyStmt);
n->relation = $3;
*************** CopyStmt: COPY opt_binary qualified_name
*** 2543,2548 ****
--- 2546,2552 ----
n->is_from = $6;
n->is_program = $7;
n->filename = $8;
+ n->exc_filename = $9;
if (n->is_program && n->filename == NULL)
ereport(ERROR,
*************** CopyStmt: COPY opt_binary qualified_name
*** 2556,2565 ****
n->options = lappend(n->options, $2);
if ($5)
n->options = lappend(n->options, $5);
! if ($9)
! n->options = lappend(n->options, $9);
! if ($11)
! n->options = list_concat(n->options, $11);
$$ = (Node *)n;
}
| COPY select_with_parens TO opt_program copy_file_name opt_with copy_options
--- 2560,2569 ----
n->options = lappend(n->options, $2);
if ($5)
n->options = lappend(n->options, $5);
! if ($10)
! n->options = lappend(n->options, $10);
! if ($12)
! n->options = list_concat(n->options, $12);
$$ = (Node *)n;
}
| COPY select_with_parens TO opt_program copy_file_name opt_with copy_options
*************** copy_file_name:
*** 2604,2609 ****
--- 2608,2618 ----
| STDOUT { $$ = NULL; }
;
+ opt_copy_exceptions:
+ EXCEPTIONS TO copy_file_name { $$ = ($3 ? $3 : ""); }
+ | /* EMPTY */ { $$ = NULL; }
+ ;
+
copy_options: copy_opt_list { $$ = $1; }
| '(' copy_generic_opt_list ')' { $$ = $2; }
;
*************** unreserved_keyword:
*** 13142,13147 ****
--- 13151,13157 ----
| ENUM_P
| ESCAPE
| EVENT
+ | EXCEPTIONS
| EXCLUDE
| EXCLUDING
| EXCLUSIVE
diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c
new file mode 100644
index 66d80b5..d308286
*** a/src/bin/psql/common.c
--- b/src/bin/psql/common.c
*************** AcceptResult(const PGresult *result)
*** 388,393 ****
--- 388,394 ----
case PGRES_EMPTY_QUERY:
case PGRES_COPY_IN:
case PGRES_COPY_OUT:
+ case PGRES_COPY_BOTH:
/* Fine, do nothing */
OK = true;
break;
*************** ProcessResult(PGresult **results)
*** 751,756 ****
--- 752,758 ----
case PGRES_COPY_OUT:
case PGRES_COPY_IN:
+ case PGRES_COPY_BOTH:
is_copy = true;
break;
*************** ProcessResult(PGresult **results)
*** 777,784 ****
SetCancelConn();
if (result_status == PGRES_COPY_OUT)
{
! if (!copystream)
copystream = pset.queryFout;
success = handleCopyOut(pset.db,
copystream,
©_result) && success;
--- 779,793 ----
SetCancelConn();
if (result_status == PGRES_COPY_OUT)
{
! /*
! * If we have the stream for exceptions, then this must be the
! * capture phase: use it.
! */
! if (pset.copyExcStream)
! copystream = pset.copyExcStream;
! else if (!copystream)
copystream = pset.queryFout;
+
success = handleCopyOut(pset.db,
copystream,
©_result) && success;
*************** ProcessResult(PGresult **results)
*** 794,800 ****
copy_result = NULL;
}
}
! else
{
if (!copystream)
copystream = pset.cur_cmd_source;
--- 803,809 ----
copy_result = NULL;
}
}
! else /* PGRES_COPY_IN or PGRES_COPY_BOTH */
{
if (!copystream)
copystream = pset.cur_cmd_source;
*************** PrintQueryResults(PGresult *results)
*** 913,918 ****
--- 922,928 ----
case PGRES_COPY_OUT:
case PGRES_COPY_IN:
+ case PGRES_COPY_BOTH:
/* nothing to do here */
success = true;
break;
diff --git a/src/bin/psql/copy.c b/src/bin/psql/copy.c
new file mode 100644
index 010a593..cc36071
*** a/src/bin/psql/copy.c
--- b/src/bin/psql/copy.c
*************** struct copy_options
*** 58,63 ****
--- 58,66 ----
bool program; /* is 'file' a program to popen? */
bool psql_inout; /* true = use psql stdin/stdout */
bool from; /* true = FROM, false = TO */
+ bool exceptions; /* has EXCEPTIONS TO */
+ char *exc_file; /* NULL = stdin/stdout */
+ bool exc_psql_inout; /* true = use psql stdout for exceptions */
};
*************** free_copy_options(struct copy_options *
*** 69,74 ****
--- 72,78 ----
free(ptr->before_tofrom);
free(ptr->after_tofrom);
free(ptr->file);
+ free(ptr->exc_file);
free(ptr);
}
*************** parse_slash_copy(const char *args)
*** 240,250 ****
expand_tilde(&result->file);
}
/* Collect the rest of the line (COPY options) */
token = strtokx(NULL, "", NULL, NULL,
0, false, false, pset.encoding);
if (token)
! result->after_tofrom = pg_strdup(token);
return result;
--- 244,302 ----
expand_tilde(&result->file);
}
+ result->after_tofrom = pg_strdup(""); /* initialize for appending */
+
+ /* check for COPY FROM ... EXCEPTIONS TO */
+ if (result->from)
+ {
+ token = strtokx(NULL, whitespace, NULL, NULL,
+ 0, false, false, pset.encoding);
+ if (token)
+ {
+ if (pg_strcasecmp(token, "exceptions") == 0)
+ {
+ result->exceptions = true;
+
+ token = strtokx(NULL, whitespace, NULL, NULL,
+ 0, false, false, pset.encoding);
+ if (!token || pg_strcasecmp(token, "to") != 0)
+ goto error;
+
+ token = strtokx(NULL, whitespace, ";", "'",
+ 0, false, false, pset.encoding);
+ if (!token)
+ goto error;
+
+ if (pg_strcasecmp(token, "stdin") == 0 ||
+ pg_strcasecmp(token, "stdout") == 0)
+ {
+ result->exc_file = NULL;
+ }
+ else if (pg_strcasecmp(token, "pstdin") == 0 ||
+ pg_strcasecmp(token, "pstdout") == 0)
+ {
+ result->exc_psql_inout = true;
+ result->exc_file = NULL;
+ }
+ else
+ {
+ strip_quotes(token, '\'', 0, pset.encoding);
+ result->exc_file = pg_strdup(token);
+ expand_tilde(&result->exc_file);
+ }
+ }
+ else
+ {
+ xstrcat(&result->after_tofrom, token);
+ }
+ }
+ }
+
/* Collect the rest of the line (COPY options) */
token = strtokx(NULL, "", NULL, NULL,
0, false, false, pset.encoding);
if (token)
! xstrcat(&result->after_tofrom, token);
return result;
*************** do_copy(const char *args)
*** 269,274 ****
--- 321,327 ----
{
PQExpBufferData query;
FILE *copystream;
+ FILE *excstream;
struct copy_options *options;
bool success;
*************** do_copy(const char *args)
*** 278,287 ****
if (!options)
return false;
! /* prepare to read or write the target file */
if (options->file && !options->program)
canonicalize_path(options->file);
if (options->from)
{
if (options->file)
--- 331,346 ----
if (!options)
return false;
! /* prepare to read or write the target file(s) */
if (options->file && !options->program)
canonicalize_path(options->file);
+ if (options->exc_file)
+ canonicalize_path(options->exc_file);
+
+ copystream = NULL;
+ excstream = NULL;
+
if (options->from)
{
if (options->file)
*************** do_copy(const char *args)
*** 294,305 ****
--- 353,394 ----
copystream = popen(options->file, PG_BINARY_R);
}
else
+ {
+ if (options->exc_file)
+ {
+ struct stat stbuf_exc;
+ struct stat stbuf_from;
+
+ /* check if both FROM and EXCEPTIONS TO are the same file */
+ if (stat(options->exc_file, &stbuf_exc) == 0 &&
+ stat(options->file, &stbuf_from) == 0 &&
+ stbuf_exc.st_dev == stbuf_from.st_dev &&
+ stbuf_exc.st_ino == stbuf_from.st_ino)
+ {
+ psql_error("COPY FROM and EXCEPTIONS TO cannot point to the same file\n");
+ free_copy_options(options);
+ return false;
+ }
+ }
copystream = fopen(options->file, PG_BINARY_R);
+ }
}
else if (!options->psql_inout)
copystream = pset.cur_cmd_source;
else
copystream = stdin;
+
+ if (options->exceptions)
+ {
+ if (options->exc_file)
+ {
+ excstream = fopen(options->exc_file, PG_BINARY_W);
+ }
+ else if (!options->exc_psql_inout)
+ excstream = pset.queryFout;
+ else
+ excstream = stdout;
+ }
}
else
{
*************** do_copy(const char *args)
*** 332,337 ****
--- 421,438 ----
else
psql_error("%s: %s\n",
options->file, strerror(errno));
+ if (options->exc_file && excstream)
+ fclose(excstream);
+ free_copy_options(options);
+ return false;
+ }
+
+ if (options->exceptions && !excstream)
+ {
+ psql_error("%s: %s\n",
+ options->exc_file, strerror(errno));
+ if (options->file)
+ fclose(copystream);
free_copy_options(options);
return false;
}
*************** do_copy(const char *args)
*** 353,358 ****
--- 454,461 ----
if (result < 0 || S_ISDIR(st.st_mode))
{
fclose(copystream);
+ if (options->exc_file && excstream)
+ fclose(excstream);
free_copy_options(options);
return false;
}
*************** do_copy(const char *args)
*** 366,378 ****
--- 469,485 ----
appendPQExpBufferStr(&query, " FROM STDIN ");
else
appendPQExpBufferStr(&query, " TO STDOUT ");
+ if (options->exceptions)
+ appendPQExpBufferStr(&query, " EXCEPTIONS TO STDOUT ");
if (options->after_tofrom)
appendPQExpBufferStr(&query, options->after_tofrom);
/* run it like a user command, but with copystream as data source/sink */
pset.copyStream = copystream;
+ pset.copyExcStream = excstream;
success = SendQuery(query.data);
pset.copyStream = NULL;
+ pset.copyExcStream = NULL;
termPQExpBuffer(&query);
if (options->file != NULL)
*************** do_copy(const char *args)
*** 410,415 ****
--- 517,530 ----
}
}
}
+ if (options->exc_file != NULL)
+ {
+ if (fclose(excstream) != 0)
+ {
+ psql_error("%s: %s\n", options->exc_file, strerror(errno));
+ success = false;
+ }
+ }
free_copy_options(options);
return success;
}
diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h
new file mode 100644
index ef24a4e..36c9300
*** a/src/bin/psql/settings.h
--- b/src/bin/psql/settings.h
*************** typedef struct _psqlSettings
*** 72,77 ****
--- 72,78 ----
bool queryFoutPipe; /* queryFout is from a popen() */
FILE *copyStream; /* Stream to read/write for \copy command */
+ FILE *copyExcStream; /* Stream to read exceptions for \copy command */
printQueryOpt popt;
diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c
new file mode 100644
index 11a159a..f1f65df
*** a/src/bin/psql/startup.c
--- b/src/bin/psql/startup.c
*************** main(int argc, char *argv[])
*** 122,127 ****
--- 122,128 ----
pset.queryFout = stdout;
pset.queryFoutPipe = false;
pset.copyStream = NULL;
+ pset.copyExcStream = NULL;
pset.cur_cmd_source = stdin;
pset.cur_cmd_interactive = false;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
new file mode 100644
index 82c926d..361cd80
*** a/src/bin/psql/tab-complete.c
--- b/src/bin/psql/tab-complete.c
*************** psql_completion(const char *text, int st
*** 2169,2175 ****
completion_charp = "";
matches = completion_matches(text, complete_from_files);
}
-
/* Handle COPY|BINARY <sth> FROM|TO filename */
else if ((pg_strcasecmp(prev4_wd, "COPY") == 0 ||
pg_strcasecmp(prev4_wd, "\\copy") == 0 ||
--- 2169,2174 ----
*************** psql_completion(const char *text, int st
*** 2178,2187 ****
pg_strcasecmp(prev2_wd, "TO") == 0))
{
static const char *const list_COPY[] =
! {"BINARY", "OIDS", "DELIMITER", "NULL", "CSV", "ENCODING", NULL};
!
COMPLETE_WITH_LIST(list_COPY);
}
/* Handle COPY|BINARY <sth> FROM|TO filename CSV */
else if (pg_strcasecmp(prev_wd, "CSV") == 0 &&
--- 2177,2193 ----
pg_strcasecmp(prev2_wd, "TO") == 0))
{
static const char *const list_COPY[] =
! {"BINARY", "OIDS", "DELIMITER", "NULL", "CSV", "ENCODING", "EXCEPTIONS TO", NULL};
COMPLETE_WITH_LIST(list_COPY);
}
+ /* If we have [COPY...] FROM <sth> EXCEPTIONS TO, complete with filename */
+ else if (pg_strcasecmp(prev4_wd, "FROM") == 0 &&
+ pg_strcasecmp(prev2_wd, "EXCEPTIONS") == 0 &&
+ pg_strcasecmp(prev_wd, "TO") == 0)
+ {
+ completion_charp = "";
+ matches = completion_matches(text, complete_from_files);
+ }
/* Handle COPY|BINARY <sth> FROM|TO filename CSV */
else if (pg_strcasecmp(prev_wd, "CSV") == 0 &&
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
new file mode 100644
index ba0f1b3..7137dfc
*** a/src/include/commands/copy.h
--- b/src/include/commands/copy.h
*************** extern Oid DoCopy(const CopyStmt *stmt,
*** 26,32 ****
extern void ProcessCopyOptions(CopyState cstate, bool is_from, List *options);
extern CopyState BeginCopyFrom(Relation rel, const char *filename,
! bool is_program, List *attnamelist, List *options);
extern void EndCopyFrom(CopyState cstate);
extern bool NextCopyFrom(CopyState cstate, ExprContext *econtext,
Datum *values, bool *nulls, Oid *tupleOid);
--- 26,33 ----
extern void ProcessCopyOptions(CopyState cstate, bool is_from, List *options);
extern CopyState BeginCopyFrom(Relation rel, const char *filename,
! bool is_program, const char *exc_filename,
! List *attnamelist, List *options);
extern void EndCopyFrom(CopyState cstate);
extern bool NextCopyFrom(CopyState cstate, ExprContext *econtext,
Datum *values, bool *nulls, Oid *tupleOid);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
new file mode 100644
index 9141c30..93f50cb
*** a/src/include/nodes/parsenodes.h
--- b/src/include/nodes/parsenodes.h
*************** typedef struct CopyStmt
*** 1513,1518 ****
--- 1513,1519 ----
bool is_from; /* TO or FROM */
bool is_program; /* is 'filename' a program to popen? */
char *filename; /* filename, or NULL for STDIN/STDOUT */
+ char *exc_filename; /* filename for exceptions or NULL, empty string for STDOUT */
List *options; /* List of DefElem nodes */
} CopyStmt;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
new file mode 100644
index e14dc9a..c44f02b
*** a/src/include/parser/kwlist.h
--- b/src/include/parser/kwlist.h
*************** PG_KEYWORD("enum", ENUM_P, UNRESERVED_KE
*** 143,148 ****
--- 143,149 ----
PG_KEYWORD("escape", ESCAPE, UNRESERVED_KEYWORD)
PG_KEYWORD("event", EVENT, UNRESERVED_KEYWORD)
PG_KEYWORD("except", EXCEPT, RESERVED_KEYWORD)
+ PG_KEYWORD("exceptions", EXCEPTIONS, UNRESERVED_KEYWORD)
PG_KEYWORD("exclude", EXCLUDE, UNRESERVED_KEYWORD)
PG_KEYWORD("excluding", EXCLUDING, UNRESERVED_KEYWORD)
PG_KEYWORD("exclusive", EXCLUSIVE, UNRESERVED_KEYWORD)
diff --git a/src/interfaces/ecpg/preproc/ecpg.addons b/src/interfaces/ecpg/preproc/ecpg.addons
new file mode 100644
index b3b36cf..0a415e6
*** a/src/interfaces/ecpg/preproc/ecpg.addons
--- b/src/interfaces/ecpg/preproc/ecpg.addons
*************** ECPG: where_or_current_clauseWHERECURREN
*** 192,198 ****
char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
$$ = cat_str(2,mm_strdup("where current of"), cursor_marker);
}
! ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listopt_oidscopy_fromopt_programcopy_file_namecopy_delimiteropt_withcopy_options addon
if (strcmp($6, "from") == 0 &&
(strcmp($7, "stdin") == 0 || strcmp($7, "stdout") == 0))
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
--- 192,198 ----
char *cursor_marker = $4[0] == ':' ? mm_strdup("$0") : $4;
$$ = cat_str(2,mm_strdup("where current of"), cursor_marker);
}
! ECPG: CopyStmtCOPYopt_binaryqualified_nameopt_column_listopt_oidscopy_fromopt_programcopy_file_nameopt_copy_exceptionscopy_delimiteropt_withcopy_options addon
if (strcmp($6, "from") == 0 &&
(strcmp($7, "stdin") == 0 || strcmp($7, "stdout") == 0))
mmerror(PARSE_ERROR, ET_WARNING, "COPY FROM STDIN is not implemented");
--
2.1.0
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
@ 2014-12-26 10:41 ` Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Pavel Stehule @ 2014-12-26 10:41 UTC (permalink / raw)
To: Alex Shulgin <[email protected]>; +Cc: pgsql-hackers
2014-12-25 22:23 GMT+01:00 Alex Shulgin <[email protected]>:
> Trent Shipley <[email protected]> writes:
>
> > On Friday 2007-12-14 16:22, Tom Lane wrote:
> >> Neil Conway <[email protected]> writes:
> >> > By modifying COPY: COPY IGNORE ERRORS or some such would instruct COPY
> >> > to drop (and log) rows that contain malformed data. That is, rows with
> >> > too many or too few columns, rows that result in constraint
> violations,
> >> > and rows containing columns where the data type's input function
> raises
> >> > an error. The last case is the only thing that would be a bit tricky
> to
> >> > implement, I think: you could use PG_TRY() around the
> InputFunctionCall,
> >> > but I guess you'd need a subtransaction to ensure that you reset your
> >> > state correctly after catching an error.
> >>
> >> Yeah. It's the subtransaction per row that's daunting --- not only the
> >> cycles spent for that, but the ensuing limitation to 4G rows imported
> >> per COPY.
> >
> > You could extend the COPY FROM syntax with a COMMIT EVERY n clause. This
> > would help with the 4G subtransaction limit. The cost to the ETL
> process is
> > that a simple rollback would not be guaranteed send the process back to
> it's
> > initial state. There are easy ways to deal with the rollback issue
> though.
> >
> > A {NO} RETRY {USING algorithm} clause might be useful. If the NO RETRY
> > option is selected then the COPY FROM can run without subtransactions
> and in
> > excess of the 4G per transaction limit. NO RETRY should be the default
> since
> > it preserves the legacy behavior of COPY FROM.
> >
> > You could have an EXCEPTIONS TO {filename|STDERR} clause. I would not
> give the
> > option of sending exceptions to a table since they are presumably
> malformed,
> > otherwise they would not be exceptions. (Users should re-process
> exception
> > files if they want an if good then table a else exception to table b ...)
> >
> > EXCEPTIONS TO and NO RETRY would be mutually exclusive.
> >
> >
> >> If we could somehow only do a subtransaction per failure, things would
> >> be much better, but I don't see how.
>
> Hello,
>
> Attached is a proof of concept patch for this TODO item. There is no
> docs yet, I just wanted to know if approach is sane.
>
> The added syntax is like the following:
>
> COPY [table] FROM [file/program/stdin] EXCEPTIONS TO [file or stdout]
>
> The way it's done it is abusing Copy Both mode and from my limited
> testing, that seems to just work. The error trapping itself is done
> using PG_TRY/PG_CATCH and can only catch formatting or before-insert
> trigger errors, no attempt is made to recover from a failed unique
> constraint, etc.
>
> Example in action:
>
> postgres=# \d test_copy2
> Table "public.test_copy2"
> Column | Type | Modifiers
> --------+---------+-----------
> id | integer |
> val | integer |
>
> postgres=# copy test_copy2 from program 'seq 3' exceptions to stdout;
> 1
> NOTICE: missing data for column "val"
> CONTEXT: COPY test_copy2, line 1: "1"
> 2
> NOTICE: missing data for column "val"
> CONTEXT: COPY test_copy2, line 2: "2"
> 3
> NOTICE: missing data for column "val"
> CONTEXT: COPY test_copy2, line 3: "3"
> NOTICE: total exceptions ignored: 3
>
> postgres=# \d test_copy1
> Table "public.test_copy1"
> Column | Type | Modifiers
> --------+---------+-----------
> id | integer | not null
>
> postgres=# set client_min_messages to warning;
> SET
> postgres=# copy test_copy1 from program 'ls /proc' exceptions to stdout;
> ...
> vmstat
> zoneinfo
> postgres=#
>
> Limited performance testing shows no significant difference between
> error-catching and plain code path. For example, timing
>
> copy test_copy1 from program 'seq 1000000' [exceptions to stdout]
>
> shows similar numbers with or without the added "exceptions to" clause.
>
> Now that I'm sending this I wonder if the original comment about the
> need for subtransaction around every loaded line still holds. Any
> example of what would be not properly rolled back by just PG_TRY?
>
this method is unsafe .. exception handlers doesn't free memory usually -
there is risk of memory leaks, source leaks
you can enforce same performance with block subtransactions - when you use
subtransaction for 1000 rows, then impact of subtransactions is minimal
when block fails, then you can use row level subtransaction - it works well
when you expect almost correct data.
Regards
Pavel
>
> Happy hacking!
> --
> Alex
>
>
>
> --
> Sent via pgsql-hackers mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
>
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
@ 2014-12-26 10:49 ` Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Pavel Stehule @ 2014-12-26 10:49 UTC (permalink / raw)
To: Alex Shulgin <[email protected]>; +Cc: pgsql-hackers
2014-12-26 11:41 GMT+01:00 Pavel Stehule <[email protected]>:
>
>
> 2014-12-25 22:23 GMT+01:00 Alex Shulgin <[email protected]>:
>
>> Trent Shipley <[email protected]> writes:
>>
>> > On Friday 2007-12-14 16:22, Tom Lane wrote:
>> >> Neil Conway <[email protected]> writes:
>> >> > By modifying COPY: COPY IGNORE ERRORS or some such would instruct
>> COPY
>> >> > to drop (and log) rows that contain malformed data. That is, rows
>> with
>> >> > too many or too few columns, rows that result in constraint
>> violations,
>> >> > and rows containing columns where the data type's input function
>> raises
>> >> > an error. The last case is the only thing that would be a bit tricky
>> to
>> >> > implement, I think: you could use PG_TRY() around the
>> InputFunctionCall,
>> >> > but I guess you'd need a subtransaction to ensure that you reset your
>> >> > state correctly after catching an error.
>> >>
>> >> Yeah. It's the subtransaction per row that's daunting --- not only the
>> >> cycles spent for that, but the ensuing limitation to 4G rows imported
>> >> per COPY.
>> >
>> > You could extend the COPY FROM syntax with a COMMIT EVERY n clause.
>> This
>> > would help with the 4G subtransaction limit. The cost to the ETL
>> process is
>> > that a simple rollback would not be guaranteed send the process back to
>> it's
>> > initial state. There are easy ways to deal with the rollback issue
>> though.
>> >
>> > A {NO} RETRY {USING algorithm} clause might be useful. If the NO RETRY
>> > option is selected then the COPY FROM can run without subtransactions
>> and in
>> > excess of the 4G per transaction limit. NO RETRY should be the default
>> since
>> > it preserves the legacy behavior of COPY FROM.
>> >
>> > You could have an EXCEPTIONS TO {filename|STDERR} clause. I would not
>> give the
>> > option of sending exceptions to a table since they are presumably
>> malformed,
>> > otherwise they would not be exceptions. (Users should re-process
>> exception
>> > files if they want an if good then table a else exception to table b
>> ...)
>> >
>> > EXCEPTIONS TO and NO RETRY would be mutually exclusive.
>> >
>> >
>> >> If we could somehow only do a subtransaction per failure, things would
>> >> be much better, but I don't see how.
>>
>> Hello,
>>
>> Attached is a proof of concept patch for this TODO item. There is no
>> docs yet, I just wanted to know if approach is sane.
>>
>> The added syntax is like the following:
>>
>> COPY [table] FROM [file/program/stdin] EXCEPTIONS TO [file or stdout]
>>
>> The way it's done it is abusing Copy Both mode and from my limited
>> testing, that seems to just work. The error trapping itself is done
>> using PG_TRY/PG_CATCH and can only catch formatting or before-insert
>> trigger errors, no attempt is made to recover from a failed unique
>> constraint, etc.
>>
>> Example in action:
>>
>> postgres=# \d test_copy2
>> Table "public.test_copy2"
>> Column | Type | Modifiers
>> --------+---------+-----------
>> id | integer |
>> val | integer |
>>
>> postgres=# copy test_copy2 from program 'seq 3' exceptions to stdout;
>> 1
>> NOTICE: missing data for column "val"
>> CONTEXT: COPY test_copy2, line 1: "1"
>> 2
>> NOTICE: missing data for column "val"
>> CONTEXT: COPY test_copy2, line 2: "2"
>> 3
>> NOTICE: missing data for column "val"
>> CONTEXT: COPY test_copy2, line 3: "3"
>> NOTICE: total exceptions ignored: 3
>>
>> postgres=# \d test_copy1
>> Table "public.test_copy1"
>> Column | Type | Modifiers
>> --------+---------+-----------
>> id | integer | not null
>>
>> postgres=# set client_min_messages to warning;
>> SET
>> postgres=# copy test_copy1 from program 'ls /proc' exceptions to stdout;
>> ...
>> vmstat
>> zoneinfo
>> postgres=#
>>
>> Limited performance testing shows no significant difference between
>> error-catching and plain code path. For example, timing
>>
>> copy test_copy1 from program 'seq 1000000' [exceptions to stdout]
>>
>> shows similar numbers with or without the added "exceptions to" clause.
>>
>> Now that I'm sending this I wonder if the original comment about the
>> need for subtransaction around every loaded line still holds. Any
>> example of what would be not properly rolled back by just PG_TRY?
>>
>
> this method is unsafe .. exception handlers doesn't free memory usually -
> there is risk of memory leaks, source leaks
>
> you can enforce same performance with block subtransactions - when you use
> subtransaction for 1000 rows, then impact of subtransactions is minimal
>
> when block fails, then you can use row level subtransaction - it works
> well when you expect almost correct data.
>
Two years ago I wrote a extension that did it - but I have not time to
finish it and push to upstream.
Regards
Pavel
>
> Regards
>
> Pavel
>
>
>>
>> Happy hacking!
>> --
>> Alex
>>
>>
>>
>> --
>> Sent via pgsql-hackers mailing list ([email protected])
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>>
>>
>
--
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachments:
[application/x-gzip] ftcopy-04.tgz (12.6K, ../../CAFj8pRD9Pi9besMvv0Yo0A4wqQ-D=cBXGRQhz_oS+BNgv_41_Q@mail.gmail.com/3-ftcopy-04.tgz)
download
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
@ 2021-12-18 08:55 ` Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2021-12-18 08:55 UTC (permalink / raw)
To: ; +Cc: pgsql-hackers
Hello.
Wrote a patch implementing COPY with ignoring errors in rows using block
subtransactions.
Syntax: COPY [table] FROM [file/stdin] WITH IGNORE_ERROS;
Examples:
CREATE TABLE check_ign_err (n int, m int, k int);
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
1 1 1
2 2 2 2
3 3 3
\.
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
SELECT * FROM check_ign_err;
n | m | k
---+---+---
1 | 1 | 1
3 | 3 | 3
(2 rows)
##################################################
TRUNCATE check_ign_err;
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
1 1 1
2 2
3 3 3
\.
WARNING: COPY check_ign_err, line 2: "2 2"
SELECT * FROM check_ign_err;
n | m | k
---+---+---
1 | 1 | 1
3 | 3 | 3
(2 rows)
##################################################
TRUNCATE check_ign_err;
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
1 1 1
2 a 2
3 3 3
\.
WARNING: COPY check_ign_err, line 2, column m: "a"
SELECT * FROM check_ign_err;
n | m | k
---+---+---
1 | 1 | 1
3 | 3 | 3
(2 rows)
Regards, Damir
пт, 10 дек. 2021 г. в 21:48, Pavel Stehule <[email protected]>:
>
>
> 2014-12-26 11:41 GMT+01:00 Pavel Stehule <[email protected]>:
>
>>
>>
>> 2014-12-25 22:23 GMT+01:00 Alex Shulgin <[email protected]>:
>>
>>> Trent Shipley <[email protected]> writes:
>>>
>>> > On Friday 2007-12-14 16:22, Tom Lane wrote:
>>> >> Neil Conway <[email protected]> writes:
>>> >> > By modifying COPY: COPY IGNORE ERRORS or some such would instruct
>>> COPY
>>> >> > to drop (and log) rows that contain malformed data. That is, rows
>>> with
>>> >> > too many or too few columns, rows that result in constraint
>>> violations,
>>> >> > and rows containing columns where the data type's input function
>>> raises
>>> >> > an error. The last case is the only thing that would be a bit
>>> tricky to
>>> >> > implement, I think: you could use PG_TRY() around the
>>> InputFunctionCall,
>>> >> > but I guess you'd need a subtransaction to ensure that you reset
>>> your
>>> >> > state correctly after catching an error.
>>> >>
>>> >> Yeah. It's the subtransaction per row that's daunting --- not only
>>> the
>>> >> cycles spent for that, but the ensuing limitation to 4G rows imported
>>> >> per COPY.
>>> >
>>> > You could extend the COPY FROM syntax with a COMMIT EVERY n clause.
>>> This
>>> > would help with the 4G subtransaction limit. The cost to the ETL
>>> process is
>>> > that a simple rollback would not be guaranteed send the process back
>>> to it's
>>> > initial state. There are easy ways to deal with the rollback issue
>>> though.
>>> >
>>> > A {NO} RETRY {USING algorithm} clause might be useful. If the NO
>>> RETRY
>>> > option is selected then the COPY FROM can run without subtransactions
>>> and in
>>> > excess of the 4G per transaction limit. NO RETRY should be the
>>> default since
>>> > it preserves the legacy behavior of COPY FROM.
>>> >
>>> > You could have an EXCEPTIONS TO {filename|STDERR} clause. I would not
>>> give the
>>> > option of sending exceptions to a table since they are presumably
>>> malformed,
>>> > otherwise they would not be exceptions. (Users should re-process
>>> exception
>>> > files if they want an if good then table a else exception to table b
>>> ...)
>>> >
>>> > EXCEPTIONS TO and NO RETRY would be mutually exclusive.
>>> >
>>> >
>>> >> If we could somehow only do a subtransaction per failure, things would
>>> >> be much better, but I don't see how.
>>>
>>> Hello,
>>>
>>> Attached is a proof of concept patch for this TODO item. There is no
>>> docs yet, I just wanted to know if approach is sane.
>>>
>>> The added syntax is like the following:
>>>
>>> COPY [table] FROM [file/program/stdin] EXCEPTIONS TO [file or stdout]
>>>
>>> The way it's done it is abusing Copy Both mode and from my limited
>>> testing, that seems to just work. The error trapping itself is done
>>> using PG_TRY/PG_CATCH and can only catch formatting or before-insert
>>> trigger errors, no attempt is made to recover from a failed unique
>>> constraint, etc.
>>>
>>> Example in action:
>>>
>>> postgres=# \d test_copy2
>>> Table "public.test_copy2"
>>> Column | Type | Modifiers
>>> --------+---------+-----------
>>> id | integer |
>>> val | integer |
>>>
>>> postgres=# copy test_copy2 from program 'seq 3' exceptions to stdout;
>>> 1
>>> NOTICE: missing data for column "val"
>>> CONTEXT: COPY test_copy2, line 1: "1"
>>> 2
>>> NOTICE: missing data for column "val"
>>> CONTEXT: COPY test_copy2, line 2: "2"
>>> 3
>>> NOTICE: missing data for column "val"
>>> CONTEXT: COPY test_copy2, line 3: "3"
>>> NOTICE: total exceptions ignored: 3
>>>
>>> postgres=# \d test_copy1
>>> Table "public.test_copy1"
>>> Column | Type | Modifiers
>>> --------+---------+-----------
>>> id | integer | not null
>>>
>>> postgres=# set client_min_messages to warning;
>>> SET
>>> postgres=# copy test_copy1 from program 'ls /proc' exceptions to stdout;
>>> ...
>>> vmstat
>>> zoneinfo
>>> postgres=#
>>>
>>> Limited performance testing shows no significant difference between
>>> error-catching and plain code path. For example, timing
>>>
>>> copy test_copy1 from program 'seq 1000000' [exceptions to stdout]
>>>
>>> shows similar numbers with or without the added "exceptions to" clause.
>>>
>>> Now that I'm sending this I wonder if the original comment about the
>>> need for subtransaction around every loaded line still holds. Any
>>> example of what would be not properly rolled back by just PG_TRY?
>>>
>>
>> this method is unsafe .. exception handlers doesn't free memory usually -
>> there is risk of memory leaks, source leaks
>>
>> you can enforce same performance with block subtransactions - when you
>> use subtransaction for 1000 rows, then impact of subtransactions is minimal
>>
>> when block fails, then you can use row level subtransaction - it works
>> well when you expect almost correct data.
>>
>
> Two years ago I wrote a extension that did it - but I have not time to
> finish it and push to upstream.
>
> Regards
>
> Pavel
>
>
>>
>> Regards
>>
>> Pavel
>>
>>
>>>
>>> Happy hacking!
>>> --
>>> Alex
>>>
>>>
>>>
>>> --
>>> Sent via pgsql-hackers mailing list ([email protected])
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgsql-hackers
>>>
>>>
>>
>
> --
> Sent via pgsql-hackers mailing list ([email protected])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
Attachments:
[text/x-patch] 0001-COPY-IGNORE_ERRORS-option.patch (35.3K, ../../CALH1LgsKDk4XthrM4m8tAqfoPyNgDCSFV3Go1nevXze-8dwKrA@mail.gmail.com/3-0001-COPY-IGNORE_ERRORS-option.patch)
download | inline diff:
From b2dbe11f103655bf845aabc3d7af3697d1441b43 Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Oct 2021 11:55:18 +0300
Subject: [PATCH] COPY IGNORE_ERRORS option
---
doc/src/sgml/ref/copy.sgml | 13 +
src/backend/commands/copy.c | 8 +
src/backend/commands/copyfrom.c | 72 ++++-
src/backend/commands/copyfromparse.c | 13 +-
src/backend/parser/gram.y | 8 +-
src/backend/utils/.gitignore | 1 -
src/backend/utils/errcodes.h | 354 +++++++++++++++++++++++
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 1 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 32 ++
src/test/regress/sql/copy2.sql | 26 ++
13 files changed, 525 insertions(+), 8 deletions(-)
create mode 100644 src/backend/utils/errcodes.h
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 4624c8f4c9..5ca8ff876d 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> ]
@@ -233,6 +234,18 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drop rows that contain malformed data while copying. That is rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows that result in constraint violations, rows containing columns where
+ the data type's input function raises an error.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 53f4853141..9ddc8c3d96 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -337,6 +337,7 @@ ProcessCopyOptions(ParseState *pstate,
{
bool format_specified = false;
bool freeze_specified = false;
+ bool ignore_errors_specified = false;
bool header_specified = false;
ListCell *option;
@@ -377,6 +378,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f366a818a1..d416222faf 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -852,9 +852,75 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ /*
+ * If option IGNORE_ERRORS is enabled, COPY skip rows with errors.
+ * NextCopyFrom() directly store the values/nulls array in the slot.
+ */
+ if (cstate->opts.ignore_errors)
+ {
+ bool break_for = false;
+ bool skip_tuple_ignore_errors = false;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
+
+ BeginInternalSubTransaction(NULL);
+ MemoryContextSwitchTo(ccxt);
+
+ PG_TRY();
+ {
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ {
+ // can't do break in PG_TRY
+ break_for = true;
+ }
+
+ ReleaseCurrentSubTransaction();
+ MemoryContextSwitchTo(ccxt);
+ CurrentResourceOwner = oldowner;
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
+ switch (errdata->sqlerrcode)
+ {
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ skip_tuple_ignore_errors = true;
+ elog(WARNING, errdata->context);
+
+ RollbackAndReleaseCurrentSubTransaction();
+ MemoryContextSwitchTo(ccxt);
+ CurrentResourceOwner = oldowner;
+
+ ExecClearTuple(myslot);
+ MemSet(myslot->tts_values, 0, cstate->attr_count * sizeof(Datum));
+ MemSet(myslot->tts_isnull, true, cstate->attr_count * sizeof(bool));
+
+ break;
+ default:
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+ }
+ PG_END_TRY();
+
+ if (break_for)
+ break;
+
+ if (skip_tuple_ignore_errors)
+ continue;
+ }
+ else
+ {
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ }
ExecStoreVirtualTuple(myslot);
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index aac10165ec..702b70861d 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -817,6 +817,7 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
tupDesc = RelationGetDescr(cstate->rel);
num_phys_attrs = tupDesc->natts;
attr_count = list_length(cstate->attnumlist);
+ cstate->attr_count = attr_count;
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
@@ -893,6 +894,7 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
string,
typioparams[m],
att->atttypmod);
+
if (string != NULL)
nulls[m] = false;
cstate->cur_attname = NULL;
@@ -1446,10 +1448,17 @@ CopyReadAttributesText(CopyFromState cstate)
if (cstate->max_fields <= 0)
{
if (cstate->line_buf.len != 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ {
+ if (cstate->opts.ignore_errors)
+ ereport(ERROR,
+ (errcode(ERRCODE_FOR_IGNORE_ERRORS),
errmsg("extra data after last expected column")));
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
return 0;
+ }
}
resetStringInfo(&cstate->attribute_buf);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 86ce33bd97..5f6863e5e1 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -674,7 +674,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3160,6 +3160,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *)makeInteger(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeInteger(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *)makeString($3), @1);
@@ -15659,6 +15663,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -16209,6 +16214,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/backend/utils/.gitignore b/src/backend/utils/.gitignore
index 0685556959..3c679d07ff 100644
--- a/src/backend/utils/.gitignore
+++ b/src/backend/utils/.gitignore
@@ -3,4 +3,3 @@
/fmgrprotos.h
/fmgr-stamp
/probes.h
-/errcodes.h
diff --git a/src/backend/utils/errcodes.h b/src/backend/utils/errcodes.h
new file mode 100644
index 0000000000..b187599bdd
--- /dev/null
+++ b/src/backend/utils/errcodes.h
@@ -0,0 +1,354 @@
+/* autogenerated from src/backend/utils/errcodes.txt, do not edit */
+/* there is deliberately not an #ifndef ERRCODES_H here */
+
+/* Class 00 - Successful Completion */
+#define ERRCODE_SUCCESSFUL_COMPLETION MAKE_SQLSTATE('0','0','0','0','0')
+
+/* Class 01 - Warning */
+#define ERRCODE_WARNING MAKE_SQLSTATE('0','1','0','0','0')
+#define ERRCODE_WARNING_DYNAMIC_RESULT_SETS_RETURNED MAKE_SQLSTATE('0','1','0','0','C')
+#define ERRCODE_WARNING_IMPLICIT_ZERO_BIT_PADDING MAKE_SQLSTATE('0','1','0','0','8')
+#define ERRCODE_WARNING_NULL_VALUE_ELIMINATED_IN_SET_FUNCTION MAKE_SQLSTATE('0','1','0','0','3')
+#define ERRCODE_WARNING_PRIVILEGE_NOT_GRANTED MAKE_SQLSTATE('0','1','0','0','7')
+#define ERRCODE_WARNING_PRIVILEGE_NOT_REVOKED MAKE_SQLSTATE('0','1','0','0','6')
+#define ERRCODE_WARNING_STRING_DATA_RIGHT_TRUNCATION MAKE_SQLSTATE('0','1','0','0','4')
+#define ERRCODE_WARNING_DEPRECATED_FEATURE MAKE_SQLSTATE('0','1','P','0','1')
+
+/* Class 02 - No Data (this is also a warning class per the SQL standard) */
+#define ERRCODE_NO_DATA MAKE_SQLSTATE('0','2','0','0','0')
+#define ERRCODE_NO_ADDITIONAL_DYNAMIC_RESULT_SETS_RETURNED MAKE_SQLSTATE('0','2','0','0','1')
+
+/* Class 03 - SQL Statement Not Yet Complete */
+#define ERRCODE_SQL_STATEMENT_NOT_YET_COMPLETE MAKE_SQLSTATE('0','3','0','0','0')
+
+/* Class 08 - Connection Exception */
+#define ERRCODE_CONNECTION_EXCEPTION MAKE_SQLSTATE('0','8','0','0','0')
+#define ERRCODE_CONNECTION_DOES_NOT_EXIST MAKE_SQLSTATE('0','8','0','0','3')
+#define ERRCODE_CONNECTION_FAILURE MAKE_SQLSTATE('0','8','0','0','6')
+#define ERRCODE_SQLCLIENT_UNABLE_TO_ESTABLISH_SQLCONNECTION MAKE_SQLSTATE('0','8','0','0','1')
+#define ERRCODE_SQLSERVER_REJECTED_ESTABLISHMENT_OF_SQLCONNECTION MAKE_SQLSTATE('0','8','0','0','4')
+#define ERRCODE_TRANSACTION_RESOLUTION_UNKNOWN MAKE_SQLSTATE('0','8','0','0','7')
+#define ERRCODE_PROTOCOL_VIOLATION MAKE_SQLSTATE('0','8','P','0','1')
+
+/* Class 09 - Triggered Action Exception */
+#define ERRCODE_TRIGGERED_ACTION_EXCEPTION MAKE_SQLSTATE('0','9','0','0','0')
+
+/* Class 0A - Feature Not Supported */
+#define ERRCODE_FEATURE_NOT_SUPPORTED MAKE_SQLSTATE('0','A','0','0','0')
+
+/* Class 0B - Invalid Transaction Initiation */
+#define ERRCODE_INVALID_TRANSACTION_INITIATION MAKE_SQLSTATE('0','B','0','0','0')
+
+/* Class 0F - Locator Exception */
+#define ERRCODE_LOCATOR_EXCEPTION MAKE_SQLSTATE('0','F','0','0','0')
+#define ERRCODE_L_E_INVALID_SPECIFICATION MAKE_SQLSTATE('0','F','0','0','1')
+
+/* Class 0L - Invalid Grantor */
+#define ERRCODE_INVALID_GRANTOR MAKE_SQLSTATE('0','L','0','0','0')
+#define ERRCODE_INVALID_GRANT_OPERATION MAKE_SQLSTATE('0','L','P','0','1')
+
+/* Class 0P - Invalid Role Specification */
+#define ERRCODE_INVALID_ROLE_SPECIFICATION MAKE_SQLSTATE('0','P','0','0','0')
+
+/* Class 0Z - Diagnostics Exception */
+#define ERRCODE_DIAGNOSTICS_EXCEPTION MAKE_SQLSTATE('0','Z','0','0','0')
+#define ERRCODE_STACKED_DIAGNOSTICS_ACCESSED_WITHOUT_ACTIVE_HANDLER MAKE_SQLSTATE('0','Z','0','0','2')
+
+/* Class 20 - Case Not Found */
+#define ERRCODE_CASE_NOT_FOUND MAKE_SQLSTATE('2','0','0','0','0')
+
+/* Class 21 - Cardinality Violation */
+#define ERRCODE_CARDINALITY_VIOLATION MAKE_SQLSTATE('2','1','0','0','0')
+
+/* Class 22 - Data Exception */
+#define ERRCODE_DATA_EXCEPTION MAKE_SQLSTATE('2','2','0','0','0')
+#define ERRCODE_ARRAY_ELEMENT_ERROR MAKE_SQLSTATE('2','2','0','2','E')
+#define ERRCODE_ARRAY_SUBSCRIPT_ERROR MAKE_SQLSTATE('2','2','0','2','E')
+#define ERRCODE_CHARACTER_NOT_IN_REPERTOIRE MAKE_SQLSTATE('2','2','0','2','1')
+#define ERRCODE_DATETIME_FIELD_OVERFLOW MAKE_SQLSTATE('2','2','0','0','8')
+#define ERRCODE_DATETIME_VALUE_OUT_OF_RANGE MAKE_SQLSTATE('2','2','0','0','8')
+#define ERRCODE_DIVISION_BY_ZERO MAKE_SQLSTATE('2','2','0','1','2')
+#define ERRCODE_ERROR_IN_ASSIGNMENT MAKE_SQLSTATE('2','2','0','0','5')
+#define ERRCODE_ESCAPE_CHARACTER_CONFLICT MAKE_SQLSTATE('2','2','0','0','B')
+#define ERRCODE_INDICATOR_OVERFLOW MAKE_SQLSTATE('2','2','0','2','2')
+#define ERRCODE_INTERVAL_FIELD_OVERFLOW MAKE_SQLSTATE('2','2','0','1','5')
+#define ERRCODE_INVALID_ARGUMENT_FOR_LOG MAKE_SQLSTATE('2','2','0','1','E')
+#define ERRCODE_INVALID_ARGUMENT_FOR_NTILE MAKE_SQLSTATE('2','2','0','1','4')
+#define ERRCODE_INVALID_ARGUMENT_FOR_NTH_VALUE MAKE_SQLSTATE('2','2','0','1','6')
+#define ERRCODE_INVALID_ARGUMENT_FOR_POWER_FUNCTION MAKE_SQLSTATE('2','2','0','1','F')
+#define ERRCODE_INVALID_ARGUMENT_FOR_WIDTH_BUCKET_FUNCTION MAKE_SQLSTATE('2','2','0','1','G')
+#define ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST MAKE_SQLSTATE('2','2','0','1','8')
+#define ERRCODE_INVALID_DATETIME_FORMAT MAKE_SQLSTATE('2','2','0','0','7')
+#define ERRCODE_INVALID_ESCAPE_CHARACTER MAKE_SQLSTATE('2','2','0','1','9')
+#define ERRCODE_INVALID_ESCAPE_OCTET MAKE_SQLSTATE('2','2','0','0','D')
+#define ERRCODE_INVALID_ESCAPE_SEQUENCE MAKE_SQLSTATE('2','2','0','2','5')
+#define ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER MAKE_SQLSTATE('2','2','P','0','6')
+#define ERRCODE_INVALID_INDICATOR_PARAMETER_VALUE MAKE_SQLSTATE('2','2','0','1','0')
+#define ERRCODE_INVALID_PARAMETER_VALUE MAKE_SQLSTATE('2','2','0','2','3')
+#define ERRCODE_INVALID_PRECEDING_OR_FOLLOWING_SIZE MAKE_SQLSTATE('2','2','0','1','3')
+#define ERRCODE_INVALID_REGULAR_EXPRESSION MAKE_SQLSTATE('2','2','0','1','B')
+#define ERRCODE_INVALID_ROW_COUNT_IN_LIMIT_CLAUSE MAKE_SQLSTATE('2','2','0','1','W')
+#define ERRCODE_INVALID_ROW_COUNT_IN_RESULT_OFFSET_CLAUSE MAKE_SQLSTATE('2','2','0','1','X')
+#define ERRCODE_INVALID_TABLESAMPLE_ARGUMENT MAKE_SQLSTATE('2','2','0','2','H')
+#define ERRCODE_INVALID_TABLESAMPLE_REPEAT MAKE_SQLSTATE('2','2','0','2','G')
+#define ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE MAKE_SQLSTATE('2','2','0','0','9')
+#define ERRCODE_INVALID_USE_OF_ESCAPE_CHARACTER MAKE_SQLSTATE('2','2','0','0','C')
+#define ERRCODE_MOST_SPECIFIC_TYPE_MISMATCH MAKE_SQLSTATE('2','2','0','0','G')
+#define ERRCODE_NULL_VALUE_NOT_ALLOWED MAKE_SQLSTATE('2','2','0','0','4')
+#define ERRCODE_NULL_VALUE_NO_INDICATOR_PARAMETER MAKE_SQLSTATE('2','2','0','0','2')
+#define ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE MAKE_SQLSTATE('2','2','0','0','3')
+#define ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED MAKE_SQLSTATE('2','2','0','0','H')
+#define ERRCODE_STRING_DATA_LENGTH_MISMATCH MAKE_SQLSTATE('2','2','0','2','6')
+#define ERRCODE_STRING_DATA_RIGHT_TRUNCATION MAKE_SQLSTATE('2','2','0','0','1')
+#define ERRCODE_SUBSTRING_ERROR MAKE_SQLSTATE('2','2','0','1','1')
+#define ERRCODE_TRIM_ERROR MAKE_SQLSTATE('2','2','0','2','7')
+#define ERRCODE_UNTERMINATED_C_STRING MAKE_SQLSTATE('2','2','0','2','4')
+#define ERRCODE_ZERO_LENGTH_CHARACTER_STRING MAKE_SQLSTATE('2','2','0','0','F')
+#define ERRCODE_FLOATING_POINT_EXCEPTION MAKE_SQLSTATE('2','2','P','0','1')
+#define ERRCODE_INVALID_TEXT_REPRESENTATION MAKE_SQLSTATE('2','2','P','0','2')
+#define ERRCODE_INVALID_BINARY_REPRESENTATION MAKE_SQLSTATE('2','2','P','0','3')
+#define ERRCODE_BAD_COPY_FILE_FORMAT MAKE_SQLSTATE('2','2','P','0','4')
+#define ERRCODE_UNTRANSLATABLE_CHARACTER MAKE_SQLSTATE('2','2','P','0','5')
+#define ERRCODE_NOT_AN_XML_DOCUMENT MAKE_SQLSTATE('2','2','0','0','L')
+#define ERRCODE_INVALID_XML_DOCUMENT MAKE_SQLSTATE('2','2','0','0','M')
+#define ERRCODE_INVALID_XML_CONTENT MAKE_SQLSTATE('2','2','0','0','N')
+#define ERRCODE_INVALID_XML_COMMENT MAKE_SQLSTATE('2','2','0','0','S')
+#define ERRCODE_INVALID_XML_PROCESSING_INSTRUCTION MAKE_SQLSTATE('2','2','0','0','T')
+#define ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE MAKE_SQLSTATE('2','2','0','3','0')
+#define ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION MAKE_SQLSTATE('2','2','0','3','1')
+#define ERRCODE_INVALID_JSON_TEXT MAKE_SQLSTATE('2','2','0','3', '2')
+#define ERRCODE_INVALID_SQL_JSON_SUBSCRIPT MAKE_SQLSTATE('2','2','0','3','3')
+#define ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM MAKE_SQLSTATE('2','2','0','3','4')
+#define ERRCODE_NO_SQL_JSON_ITEM MAKE_SQLSTATE('2','2','0','3','5')
+#define ERRCODE_NON_NUMERIC_SQL_JSON_ITEM MAKE_SQLSTATE('2','2','0','3','6')
+#define ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT MAKE_SQLSTATE('2','2','0','3','7')
+#define ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED MAKE_SQLSTATE('2','2','0','3','8')
+#define ERRCODE_SQL_JSON_ARRAY_NOT_FOUND MAKE_SQLSTATE('2','2','0','3','9')
+#define ERRCODE_SQL_JSON_MEMBER_NOT_FOUND MAKE_SQLSTATE('2','2','0','3','A')
+#define ERRCODE_SQL_JSON_NUMBER_NOT_FOUND MAKE_SQLSTATE('2','2','0','3','B')
+#define ERRCODE_SQL_JSON_OBJECT_NOT_FOUND MAKE_SQLSTATE('2','2','0','3','C')
+#define ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS MAKE_SQLSTATE('2','2','0','3','D')
+#define ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS MAKE_SQLSTATE('2','2','0','3','E')
+#define ERRCODE_SQL_JSON_SCALAR_REQUIRED MAKE_SQLSTATE('2','2','0','3','F')
+#define ERRCODE_FOR_IGNORE_ERRORS MAKE_SQLSTATE('2','2','0','4','0')
+
+/* Class 23 - Integrity Constraint Violation */
+#define ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION MAKE_SQLSTATE('2','3','0','0','0')
+#define ERRCODE_RESTRICT_VIOLATION MAKE_SQLSTATE('2','3','0','0','1')
+#define ERRCODE_NOT_NULL_VIOLATION MAKE_SQLSTATE('2','3','5','0','2')
+#define ERRCODE_FOREIGN_KEY_VIOLATION MAKE_SQLSTATE('2','3','5','0','3')
+#define ERRCODE_UNIQUE_VIOLATION MAKE_SQLSTATE('2','3','5','0','5')
+#define ERRCODE_CHECK_VIOLATION MAKE_SQLSTATE('2','3','5','1','4')
+#define ERRCODE_EXCLUSION_VIOLATION MAKE_SQLSTATE('2','3','P','0','1')
+
+/* Class 24 - Invalid Cursor State */
+#define ERRCODE_INVALID_CURSOR_STATE MAKE_SQLSTATE('2','4','0','0','0')
+
+/* Class 25 - Invalid Transaction State */
+#define ERRCODE_INVALID_TRANSACTION_STATE MAKE_SQLSTATE('2','5','0','0','0')
+#define ERRCODE_ACTIVE_SQL_TRANSACTION MAKE_SQLSTATE('2','5','0','0','1')
+#define ERRCODE_BRANCH_TRANSACTION_ALREADY_ACTIVE MAKE_SQLSTATE('2','5','0','0','2')
+#define ERRCODE_HELD_CURSOR_REQUIRES_SAME_ISOLATION_LEVEL MAKE_SQLSTATE('2','5','0','0','8')
+#define ERRCODE_INAPPROPRIATE_ACCESS_MODE_FOR_BRANCH_TRANSACTION MAKE_SQLSTATE('2','5','0','0','3')
+#define ERRCODE_INAPPROPRIATE_ISOLATION_LEVEL_FOR_BRANCH_TRANSACTION MAKE_SQLSTATE('2','5','0','0','4')
+#define ERRCODE_NO_ACTIVE_SQL_TRANSACTION_FOR_BRANCH_TRANSACTION MAKE_SQLSTATE('2','5','0','0','5')
+#define ERRCODE_READ_ONLY_SQL_TRANSACTION MAKE_SQLSTATE('2','5','0','0','6')
+#define ERRCODE_SCHEMA_AND_DATA_STATEMENT_MIXING_NOT_SUPPORTED MAKE_SQLSTATE('2','5','0','0','7')
+#define ERRCODE_NO_ACTIVE_SQL_TRANSACTION MAKE_SQLSTATE('2','5','P','0','1')
+#define ERRCODE_IN_FAILED_SQL_TRANSACTION MAKE_SQLSTATE('2','5','P','0','2')
+#define ERRCODE_IDLE_IN_TRANSACTION_SESSION_TIMEOUT MAKE_SQLSTATE('2','5','P','0','3')
+
+/* Class 26 - Invalid SQL Statement Name */
+#define ERRCODE_INVALID_SQL_STATEMENT_NAME MAKE_SQLSTATE('2','6','0','0','0')
+
+/* Class 27 - Triggered Data Change Violation */
+#define ERRCODE_TRIGGERED_DATA_CHANGE_VIOLATION MAKE_SQLSTATE('2','7','0','0','0')
+
+/* Class 28 - Invalid Authorization Specification */
+#define ERRCODE_INVALID_AUTHORIZATION_SPECIFICATION MAKE_SQLSTATE('2','8','0','0','0')
+#define ERRCODE_INVALID_PASSWORD MAKE_SQLSTATE('2','8','P','0','1')
+
+/* Class 2B - Dependent Privilege Descriptors Still Exist */
+#define ERRCODE_DEPENDENT_PRIVILEGE_DESCRIPTORS_STILL_EXIST MAKE_SQLSTATE('2','B','0','0','0')
+#define ERRCODE_DEPENDENT_OBJECTS_STILL_EXIST MAKE_SQLSTATE('2','B','P','0','1')
+
+/* Class 2D - Invalid Transaction Termination */
+#define ERRCODE_INVALID_TRANSACTION_TERMINATION MAKE_SQLSTATE('2','D','0','0','0')
+
+/* Class 2F - SQL Routine Exception */
+#define ERRCODE_SQL_ROUTINE_EXCEPTION MAKE_SQLSTATE('2','F','0','0','0')
+#define ERRCODE_S_R_E_FUNCTION_EXECUTED_NO_RETURN_STATEMENT MAKE_SQLSTATE('2','F','0','0','5')
+#define ERRCODE_S_R_E_MODIFYING_SQL_DATA_NOT_PERMITTED MAKE_SQLSTATE('2','F','0','0','2')
+#define ERRCODE_S_R_E_PROHIBITED_SQL_STATEMENT_ATTEMPTED MAKE_SQLSTATE('2','F','0','0','3')
+#define ERRCODE_S_R_E_READING_SQL_DATA_NOT_PERMITTED MAKE_SQLSTATE('2','F','0','0','4')
+
+/* Class 34 - Invalid Cursor Name */
+#define ERRCODE_INVALID_CURSOR_NAME MAKE_SQLSTATE('3','4','0','0','0')
+
+/* Class 38 - External Routine Exception */
+#define ERRCODE_EXTERNAL_ROUTINE_EXCEPTION MAKE_SQLSTATE('3','8','0','0','0')
+#define ERRCODE_E_R_E_CONTAINING_SQL_NOT_PERMITTED MAKE_SQLSTATE('3','8','0','0','1')
+#define ERRCODE_E_R_E_MODIFYING_SQL_DATA_NOT_PERMITTED MAKE_SQLSTATE('3','8','0','0','2')
+#define ERRCODE_E_R_E_PROHIBITED_SQL_STATEMENT_ATTEMPTED MAKE_SQLSTATE('3','8','0','0','3')
+#define ERRCODE_E_R_E_READING_SQL_DATA_NOT_PERMITTED MAKE_SQLSTATE('3','8','0','0','4')
+
+/* Class 39 - External Routine Invocation Exception */
+#define ERRCODE_EXTERNAL_ROUTINE_INVOCATION_EXCEPTION MAKE_SQLSTATE('3','9','0','0','0')
+#define ERRCODE_E_R_I_E_INVALID_SQLSTATE_RETURNED MAKE_SQLSTATE('3','9','0','0','1')
+#define ERRCODE_E_R_I_E_NULL_VALUE_NOT_ALLOWED MAKE_SQLSTATE('3','9','0','0','4')
+#define ERRCODE_E_R_I_E_TRIGGER_PROTOCOL_VIOLATED MAKE_SQLSTATE('3','9','P','0','1')
+#define ERRCODE_E_R_I_E_SRF_PROTOCOL_VIOLATED MAKE_SQLSTATE('3','9','P','0','2')
+#define ERRCODE_E_R_I_E_EVENT_TRIGGER_PROTOCOL_VIOLATED MAKE_SQLSTATE('3','9','P','0','3')
+
+/* Class 3B - Savepoint Exception */
+#define ERRCODE_SAVEPOINT_EXCEPTION MAKE_SQLSTATE('3','B','0','0','0')
+#define ERRCODE_S_E_INVALID_SPECIFICATION MAKE_SQLSTATE('3','B','0','0','1')
+
+/* Class 3D - Invalid Catalog Name */
+#define ERRCODE_INVALID_CATALOG_NAME MAKE_SQLSTATE('3','D','0','0','0')
+
+/* Class 3F - Invalid Schema Name */
+#define ERRCODE_INVALID_SCHEMA_NAME MAKE_SQLSTATE('3','F','0','0','0')
+
+/* Class 40 - Transaction Rollback */
+#define ERRCODE_TRANSACTION_ROLLBACK MAKE_SQLSTATE('4','0','0','0','0')
+#define ERRCODE_T_R_INTEGRITY_CONSTRAINT_VIOLATION MAKE_SQLSTATE('4','0','0','0','2')
+#define ERRCODE_T_R_SERIALIZATION_FAILURE MAKE_SQLSTATE('4','0','0','0','1')
+#define ERRCODE_T_R_STATEMENT_COMPLETION_UNKNOWN MAKE_SQLSTATE('4','0','0','0','3')
+#define ERRCODE_T_R_DEADLOCK_DETECTED MAKE_SQLSTATE('4','0','P','0','1')
+
+/* Class 42 - Syntax Error or Access Rule Violation */
+#define ERRCODE_SYNTAX_ERROR_OR_ACCESS_RULE_VIOLATION MAKE_SQLSTATE('4','2','0','0','0')
+#define ERRCODE_SYNTAX_ERROR MAKE_SQLSTATE('4','2','6','0','1')
+#define ERRCODE_INSUFFICIENT_PRIVILEGE MAKE_SQLSTATE('4','2','5','0','1')
+#define ERRCODE_CANNOT_COERCE MAKE_SQLSTATE('4','2','8','4','6')
+#define ERRCODE_GROUPING_ERROR MAKE_SQLSTATE('4','2','8','0','3')
+#define ERRCODE_WINDOWING_ERROR MAKE_SQLSTATE('4','2','P','2','0')
+#define ERRCODE_INVALID_RECURSION MAKE_SQLSTATE('4','2','P','1','9')
+#define ERRCODE_INVALID_FOREIGN_KEY MAKE_SQLSTATE('4','2','8','3','0')
+#define ERRCODE_INVALID_NAME MAKE_SQLSTATE('4','2','6','0','2')
+#define ERRCODE_NAME_TOO_LONG MAKE_SQLSTATE('4','2','6','2','2')
+#define ERRCODE_RESERVED_NAME MAKE_SQLSTATE('4','2','9','3','9')
+#define ERRCODE_DATATYPE_MISMATCH MAKE_SQLSTATE('4','2','8','0','4')
+#define ERRCODE_INDETERMINATE_DATATYPE MAKE_SQLSTATE('4','2','P','1','8')
+#define ERRCODE_COLLATION_MISMATCH MAKE_SQLSTATE('4','2','P','2','1')
+#define ERRCODE_INDETERMINATE_COLLATION MAKE_SQLSTATE('4','2','P','2','2')
+#define ERRCODE_WRONG_OBJECT_TYPE MAKE_SQLSTATE('4','2','8','0','9')
+#define ERRCODE_GENERATED_ALWAYS MAKE_SQLSTATE('4','2','8','C','9')
+#define ERRCODE_UNDEFINED_COLUMN MAKE_SQLSTATE('4','2','7','0','3')
+#define ERRCODE_UNDEFINED_CURSOR MAKE_SQLSTATE('3','4','0','0','0')
+#define ERRCODE_UNDEFINED_DATABASE MAKE_SQLSTATE('3','D','0','0','0')
+#define ERRCODE_UNDEFINED_FUNCTION MAKE_SQLSTATE('4','2','8','8','3')
+#define ERRCODE_UNDEFINED_PSTATEMENT MAKE_SQLSTATE('2','6','0','0','0')
+#define ERRCODE_UNDEFINED_SCHEMA MAKE_SQLSTATE('3','F','0','0','0')
+#define ERRCODE_UNDEFINED_TABLE MAKE_SQLSTATE('4','2','P','0','1')
+#define ERRCODE_UNDEFINED_PARAMETER MAKE_SQLSTATE('4','2','P','0','2')
+#define ERRCODE_UNDEFINED_OBJECT MAKE_SQLSTATE('4','2','7','0','4')
+#define ERRCODE_DUPLICATE_COLUMN MAKE_SQLSTATE('4','2','7','0','1')
+#define ERRCODE_DUPLICATE_CURSOR MAKE_SQLSTATE('4','2','P','0','3')
+#define ERRCODE_DUPLICATE_DATABASE MAKE_SQLSTATE('4','2','P','0','4')
+#define ERRCODE_DUPLICATE_FUNCTION MAKE_SQLSTATE('4','2','7','2','3')
+#define ERRCODE_DUPLICATE_PSTATEMENT MAKE_SQLSTATE('4','2','P','0','5')
+#define ERRCODE_DUPLICATE_SCHEMA MAKE_SQLSTATE('4','2','P','0','6')
+#define ERRCODE_DUPLICATE_TABLE MAKE_SQLSTATE('4','2','P','0','7')
+#define ERRCODE_DUPLICATE_ALIAS MAKE_SQLSTATE('4','2','7','1','2')
+#define ERRCODE_DUPLICATE_OBJECT MAKE_SQLSTATE('4','2','7','1','0')
+#define ERRCODE_AMBIGUOUS_COLUMN MAKE_SQLSTATE('4','2','7','0','2')
+#define ERRCODE_AMBIGUOUS_FUNCTION MAKE_SQLSTATE('4','2','7','2','5')
+#define ERRCODE_AMBIGUOUS_PARAMETER MAKE_SQLSTATE('4','2','P','0','8')
+#define ERRCODE_AMBIGUOUS_ALIAS MAKE_SQLSTATE('4','2','P','0','9')
+#define ERRCODE_INVALID_COLUMN_REFERENCE MAKE_SQLSTATE('4','2','P','1','0')
+#define ERRCODE_INVALID_COLUMN_DEFINITION MAKE_SQLSTATE('4','2','6','1','1')
+#define ERRCODE_INVALID_CURSOR_DEFINITION MAKE_SQLSTATE('4','2','P','1','1')
+#define ERRCODE_INVALID_DATABASE_DEFINITION MAKE_SQLSTATE('4','2','P','1','2')
+#define ERRCODE_INVALID_FUNCTION_DEFINITION MAKE_SQLSTATE('4','2','P','1','3')
+#define ERRCODE_INVALID_PSTATEMENT_DEFINITION MAKE_SQLSTATE('4','2','P','1','4')
+#define ERRCODE_INVALID_SCHEMA_DEFINITION MAKE_SQLSTATE('4','2','P','1','5')
+#define ERRCODE_INVALID_TABLE_DEFINITION MAKE_SQLSTATE('4','2','P','1','6')
+#define ERRCODE_INVALID_OBJECT_DEFINITION MAKE_SQLSTATE('4','2','P','1','7')
+
+/* Class 44 - WITH CHECK OPTION Violation */
+#define ERRCODE_WITH_CHECK_OPTION_VIOLATION MAKE_SQLSTATE('4','4','0','0','0')
+
+/* Class 53 - Insufficient Resources */
+#define ERRCODE_INSUFFICIENT_RESOURCES MAKE_SQLSTATE('5','3','0','0','0')
+#define ERRCODE_DISK_FULL MAKE_SQLSTATE('5','3','1','0','0')
+#define ERRCODE_OUT_OF_MEMORY MAKE_SQLSTATE('5','3','2','0','0')
+#define ERRCODE_TOO_MANY_CONNECTIONS MAKE_SQLSTATE('5','3','3','0','0')
+#define ERRCODE_CONFIGURATION_LIMIT_EXCEEDED MAKE_SQLSTATE('5','3','4','0','0')
+
+/* Class 54 - Program Limit Exceeded */
+#define ERRCODE_PROGRAM_LIMIT_EXCEEDED MAKE_SQLSTATE('5','4','0','0','0')
+#define ERRCODE_STATEMENT_TOO_COMPLEX MAKE_SQLSTATE('5','4','0','0','1')
+#define ERRCODE_TOO_MANY_COLUMNS MAKE_SQLSTATE('5','4','0','1','1')
+#define ERRCODE_TOO_MANY_ARGUMENTS MAKE_SQLSTATE('5','4','0','2','3')
+
+/* Class 55 - Object Not In Prerequisite State */
+#define ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE MAKE_SQLSTATE('5','5','0','0','0')
+#define ERRCODE_OBJECT_IN_USE MAKE_SQLSTATE('5','5','0','0','6')
+#define ERRCODE_CANT_CHANGE_RUNTIME_PARAM MAKE_SQLSTATE('5','5','P','0','2')
+#define ERRCODE_LOCK_NOT_AVAILABLE MAKE_SQLSTATE('5','5','P','0','3')
+#define ERRCODE_UNSAFE_NEW_ENUM_VALUE_USAGE MAKE_SQLSTATE('5','5','P','0','4')
+
+/* Class 57 - Operator Intervention */
+#define ERRCODE_OPERATOR_INTERVENTION MAKE_SQLSTATE('5','7','0','0','0')
+#define ERRCODE_QUERY_CANCELED MAKE_SQLSTATE('5','7','0','1','4')
+#define ERRCODE_ADMIN_SHUTDOWN MAKE_SQLSTATE('5','7','P','0','1')
+#define ERRCODE_CRASH_SHUTDOWN MAKE_SQLSTATE('5','7','P','0','2')
+#define ERRCODE_CANNOT_CONNECT_NOW MAKE_SQLSTATE('5','7','P','0','3')
+#define ERRCODE_DATABASE_DROPPED MAKE_SQLSTATE('5','7','P','0','4')
+#define ERRCODE_IDLE_SESSION_TIMEOUT MAKE_SQLSTATE('5','7','P','0','5')
+
+/* Class 58 - System Error (errors external to PostgreSQL itself) */
+#define ERRCODE_SYSTEM_ERROR MAKE_SQLSTATE('5','8','0','0','0')
+#define ERRCODE_IO_ERROR MAKE_SQLSTATE('5','8','0','3','0')
+#define ERRCODE_UNDEFINED_FILE MAKE_SQLSTATE('5','8','P','0','1')
+#define ERRCODE_DUPLICATE_FILE MAKE_SQLSTATE('5','8','P','0','2')
+
+/* Class 72 - Snapshot Failure */
+#define ERRCODE_SNAPSHOT_TOO_OLD MAKE_SQLSTATE('7','2','0','0','0')
+
+/* Class F0 - Configuration File Error */
+#define ERRCODE_CONFIG_FILE_ERROR MAKE_SQLSTATE('F','0','0','0','0')
+#define ERRCODE_LOCK_FILE_EXISTS MAKE_SQLSTATE('F','0','0','0','1')
+
+/* Class HV - Foreign Data Wrapper Error (SQL/MED) */
+#define ERRCODE_FDW_ERROR MAKE_SQLSTATE('H','V','0','0','0')
+#define ERRCODE_FDW_COLUMN_NAME_NOT_FOUND MAKE_SQLSTATE('H','V','0','0','5')
+#define ERRCODE_FDW_DYNAMIC_PARAMETER_VALUE_NEEDED MAKE_SQLSTATE('H','V','0','0','2')
+#define ERRCODE_FDW_FUNCTION_SEQUENCE_ERROR MAKE_SQLSTATE('H','V','0','1','0')
+#define ERRCODE_FDW_INCONSISTENT_DESCRIPTOR_INFORMATION MAKE_SQLSTATE('H','V','0','2','1')
+#define ERRCODE_FDW_INVALID_ATTRIBUTE_VALUE MAKE_SQLSTATE('H','V','0','2','4')
+#define ERRCODE_FDW_INVALID_COLUMN_NAME MAKE_SQLSTATE('H','V','0','0','7')
+#define ERRCODE_FDW_INVALID_COLUMN_NUMBER MAKE_SQLSTATE('H','V','0','0','8')
+#define ERRCODE_FDW_INVALID_DATA_TYPE MAKE_SQLSTATE('H','V','0','0','4')
+#define ERRCODE_FDW_INVALID_DATA_TYPE_DESCRIPTORS MAKE_SQLSTATE('H','V','0','0','6')
+#define ERRCODE_FDW_INVALID_DESCRIPTOR_FIELD_IDENTIFIER MAKE_SQLSTATE('H','V','0','9','1')
+#define ERRCODE_FDW_INVALID_HANDLE MAKE_SQLSTATE('H','V','0','0','B')
+#define ERRCODE_FDW_INVALID_OPTION_INDEX MAKE_SQLSTATE('H','V','0','0','C')
+#define ERRCODE_FDW_INVALID_OPTION_NAME MAKE_SQLSTATE('H','V','0','0','D')
+#define ERRCODE_FDW_INVALID_STRING_LENGTH_OR_BUFFER_LENGTH MAKE_SQLSTATE('H','V','0','9','0')
+#define ERRCODE_FDW_INVALID_STRING_FORMAT MAKE_SQLSTATE('H','V','0','0','A')
+#define ERRCODE_FDW_INVALID_USE_OF_NULL_POINTER MAKE_SQLSTATE('H','V','0','0','9')
+#define ERRCODE_FDW_TOO_MANY_HANDLES MAKE_SQLSTATE('H','V','0','1','4')
+#define ERRCODE_FDW_OUT_OF_MEMORY MAKE_SQLSTATE('H','V','0','0','1')
+#define ERRCODE_FDW_NO_SCHEMAS MAKE_SQLSTATE('H','V','0','0','P')
+#define ERRCODE_FDW_OPTION_NAME_NOT_FOUND MAKE_SQLSTATE('H','V','0','0','J')
+#define ERRCODE_FDW_REPLY_HANDLE MAKE_SQLSTATE('H','V','0','0','K')
+#define ERRCODE_FDW_SCHEMA_NOT_FOUND MAKE_SQLSTATE('H','V','0','0','Q')
+#define ERRCODE_FDW_TABLE_NOT_FOUND MAKE_SQLSTATE('H','V','0','0','R')
+#define ERRCODE_FDW_UNABLE_TO_CREATE_EXECUTION MAKE_SQLSTATE('H','V','0','0','L')
+#define ERRCODE_FDW_UNABLE_TO_CREATE_REPLY MAKE_SQLSTATE('H','V','0','0','M')
+#define ERRCODE_FDW_UNABLE_TO_ESTABLISH_CONNECTION MAKE_SQLSTATE('H','V','0','0','N')
+
+/* Class P0 - PL/pgSQL Error */
+#define ERRCODE_PLPGSQL_ERROR MAKE_SQLSTATE('P','0','0','0','0')
+#define ERRCODE_RAISE_EXCEPTION MAKE_SQLSTATE('P','0','0','0','1')
+#define ERRCODE_NO_DATA_FOUND MAKE_SQLSTATE('P','0','0','0','2')
+#define ERRCODE_TOO_MANY_ROWS MAKE_SQLSTATE('P','0','0','0','3')
+#define ERRCODE_ASSERT_FAILURE MAKE_SQLSTATE('P','0','0','0','4')
+
+/* Class XX - Internal Error */
+#define ERRCODE_INTERNAL_ERROR MAKE_SQLSTATE('X','X','0','0','0')
+#define ERRCODE_DATA_CORRUPTED MAKE_SQLSTATE('X','X','0','0','1')
+#define ERRCODE_INDEX_CORRUPTED MAKE_SQLSTATE('X','X','0','0','2')
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 2f412ca3db..855cbeb401 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2545,7 +2545,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 264895d278..ade74ce5a1 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -31,6 +31,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
bool header_line; /* CSV header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 4d68d9cceb..93a3b74999 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -68,6 +68,7 @@ typedef struct CopyFromStateData
/* parameters from the COPY command */
Relation rel; /* relation to copy from */
List *attnumlist; /* integer list of attnums to copy */
+ int attr_count; /* length of attnumlist */
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f836acf876..2bb3cccea9 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -196,6 +196,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..4e04efcbba 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,37 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- test IGNORE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 3 | 3 | 3
+(2 rows)
+
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2"
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 3 | 3 | 3
+(2 rows)
+
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2, column m: "a"
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 3 | 3 | 3
+(2 rows)
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -663,3 +694,4 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..3642c11f91 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,31 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- test IGNORE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3 3
+\.
+SELECT * FROM check_ign_err;
+
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2
+3 3 3
+\.
+SELECT * FROM check_ign_err;
+
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 a 2
+3 3 3
+\.
+SELECT * FROM check_ign_err;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -468,3 +493,4 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
--
2.25.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2021-12-19 05:09 ` Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Pavel Stehule @ 2021-12-19 05:09 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers
Hi
so 18. 12. 2021 v 9:55 odesílatel Damir Belyalov <[email protected]>
napsal:
> Hello.
>
> Wrote a patch implementing COPY with ignoring errors in rows using block
> subtransactions.
>
It is great so you are working on this patch. Unfortunately, I am afraid
this simple design is not optimal. Using subtransaction for every row has
too big overhead. I think it should use subtransaction for blocks of rows
(1000 rows), and only when there is an exception, then it should replay
inserts in subtransaction per row. You should check performance overhead.
Regards
Pavel
> Syntax: COPY [table] FROM [file/stdin] WITH IGNORE_ERROS;
>
> Examples:
> CREATE TABLE check_ign_err (n int, m int, k int);
> COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
> 1 1 1
> 2 2 2 2
> 3 3 3
> \.
> WARNING: COPY check_ign_err, line 2: "2 2 2 2"
> SELECT * FROM check_ign_err;
> n | m | k
> ---+---+---
> 1 | 1 | 1
> 3 | 3 | 3
> (2 rows)
>
> ##################################################
>
> TRUNCATE check_ign_err;
> COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
> 1 1 1
> 2 2
> 3 3 3
> \.
> WARNING: COPY check_ign_err, line 2: "2 2"
> SELECT * FROM check_ign_err;
> n | m | k
> ---+---+---
> 1 | 1 | 1
> 3 | 3 | 3
> (2 rows)
>
> ##################################################
>
> TRUNCATE check_ign_err;
> COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
> 1 1 1
> 2 a 2
> 3 3 3
> \.
> WARNING: COPY check_ign_err, line 2, column m: "a"
> SELECT * FROM check_ign_err;
> n | m | k
> ---+---+---
> 1 | 1 | 1
> 3 | 3 | 3
> (2 rows)
>
>
>
> Regards, Damir
>
> пт, 10 дек. 2021 г. в 21:48, Pavel Stehule <[email protected]>:
>
>>
>>
>> 2014-12-26 11:41 GMT+01:00 Pavel Stehule <[email protected]>:
>>
>>>
>>>
>>> 2014-12-25 22:23 GMT+01:00 Alex Shulgin <[email protected]>:
>>>
>>>> Trent Shipley <[email protected]> writes:
>>>>
>>>> > On Friday 2007-12-14 16:22, Tom Lane wrote:
>>>> >> Neil Conway <[email protected]> writes:
>>>> >> > By modifying COPY: COPY IGNORE ERRORS or some such would instruct
>>>> COPY
>>>> >> > to drop (and log) rows that contain malformed data. That is, rows
>>>> with
>>>> >> > too many or too few columns, rows that result in constraint
>>>> violations,
>>>> >> > and rows containing columns where the data type's input function
>>>> raises
>>>> >> > an error. The last case is the only thing that would be a bit
>>>> tricky to
>>>> >> > implement, I think: you could use PG_TRY() around the
>>>> InputFunctionCall,
>>>> >> > but I guess you'd need a subtransaction to ensure that you reset
>>>> your
>>>> >> > state correctly after catching an error.
>>>> >>
>>>> >> Yeah. It's the subtransaction per row that's daunting --- not only
>>>> the
>>>> >> cycles spent for that, but the ensuing limitation to 4G rows imported
>>>> >> per COPY.
>>>> >
>>>> > You could extend the COPY FROM syntax with a COMMIT EVERY n clause.
>>>> This
>>>> > would help with the 4G subtransaction limit. The cost to the ETL
>>>> process is
>>>> > that a simple rollback would not be guaranteed send the process back
>>>> to it's
>>>> > initial state. There are easy ways to deal with the rollback issue
>>>> though.
>>>> >
>>>> > A {NO} RETRY {USING algorithm} clause might be useful. If the NO
>>>> RETRY
>>>> > option is selected then the COPY FROM can run without subtransactions
>>>> and in
>>>> > excess of the 4G per transaction limit. NO RETRY should be the
>>>> default since
>>>> > it preserves the legacy behavior of COPY FROM.
>>>> >
>>>> > You could have an EXCEPTIONS TO {filename|STDERR} clause. I would not
>>>> give the
>>>> > option of sending exceptions to a table since they are presumably
>>>> malformed,
>>>> > otherwise they would not be exceptions. (Users should re-process
>>>> exception
>>>> > files if they want an if good then table a else exception to table b
>>>> ...)
>>>> >
>>>> > EXCEPTIONS TO and NO RETRY would be mutually exclusive.
>>>> >
>>>> >
>>>> >> If we could somehow only do a subtransaction per failure, things
>>>> would
>>>> >> be much better, but I don't see how.
>>>>
>>>> Hello,
>>>>
>>>> Attached is a proof of concept patch for this TODO item. There is no
>>>> docs yet, I just wanted to know if approach is sane.
>>>>
>>>> The added syntax is like the following:
>>>>
>>>> COPY [table] FROM [file/program/stdin] EXCEPTIONS TO [file or stdout]
>>>>
>>>> The way it's done it is abusing Copy Both mode and from my limited
>>>> testing, that seems to just work. The error trapping itself is done
>>>> using PG_TRY/PG_CATCH and can only catch formatting or before-insert
>>>> trigger errors, no attempt is made to recover from a failed unique
>>>> constraint, etc.
>>>>
>>>> Example in action:
>>>>
>>>> postgres=# \d test_copy2
>>>> Table "public.test_copy2"
>>>> Column | Type | Modifiers
>>>> --------+---------+-----------
>>>> id | integer |
>>>> val | integer |
>>>>
>>>> postgres=# copy test_copy2 from program 'seq 3' exceptions to stdout;
>>>> 1
>>>> NOTICE: missing data for column "val"
>>>> CONTEXT: COPY test_copy2, line 1: "1"
>>>> 2
>>>> NOTICE: missing data for column "val"
>>>> CONTEXT: COPY test_copy2, line 2: "2"
>>>> 3
>>>> NOTICE: missing data for column "val"
>>>> CONTEXT: COPY test_copy2, line 3: "3"
>>>> NOTICE: total exceptions ignored: 3
>>>>
>>>> postgres=# \d test_copy1
>>>> Table "public.test_copy1"
>>>> Column | Type | Modifiers
>>>> --------+---------+-----------
>>>> id | integer | not null
>>>>
>>>> postgres=# set client_min_messages to warning;
>>>> SET
>>>> postgres=# copy test_copy1 from program 'ls /proc' exceptions to stdout;
>>>> ...
>>>> vmstat
>>>> zoneinfo
>>>> postgres=#
>>>>
>>>> Limited performance testing shows no significant difference between
>>>> error-catching and plain code path. For example, timing
>>>>
>>>> copy test_copy1 from program 'seq 1000000' [exceptions to stdout]
>>>>
>>>> shows similar numbers with or without the added "exceptions to" clause.
>>>>
>>>> Now that I'm sending this I wonder if the original comment about the
>>>> need for subtransaction around every loaded line still holds. Any
>>>> example of what would be not properly rolled back by just PG_TRY?
>>>>
>>>
>>> this method is unsafe .. exception handlers doesn't free memory usually
>>> - there is risk of memory leaks, source leaks
>>>
>>> you can enforce same performance with block subtransactions - when you
>>> use subtransaction for 1000 rows, then impact of subtransactions is minimal
>>>
>>> when block fails, then you can use row level subtransaction - it works
>>> well when you expect almost correct data.
>>>
>>
>> Two years ago I wrote a extension that did it - but I have not time to
>> finish it and push to upstream.
>>
>> Regards
>>
>> Pavel
>>
>>
>>>
>>> Regards
>>>
>>> Pavel
>>>
>>>
>>>>
>>>> Happy hacking!
>>>> --
>>>> Alex
>>>>
>>>>
>>>>
>>>> --
>>>> Sent via pgsql-hackers mailing list ([email protected])
>>>> To make changes to your subscription:
>>>> http://www.postgresql.org/mailpref/pgsql-hackers
>>>>
>>>>
>>>
>>
>> --
>> Sent via pgsql-hackers mailing list ([email protected])
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>>
>
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
@ 2022-07-19 12:40 ` Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-07-19 12:40 UTC (permalink / raw)
To: pgsql-hackers; [email protected]
Hi!
Improved my patch by adding block subtransactions.
The block size is determined by the REPLAY_BUFFER_SIZE parameter.
I used the idea of a buffer for accumulating tuples in it.
If we read REPLAY_BUFFER_SIZE rows without errors, the subtransaction will
be committed.
If we find an error, the subtransaction will rollback and the buffer will
be replayed containing tuples.
In the patch REPLAY_BUFFER_SIZE equals 3, but it can be changed to any
other number (for example 1000).
There is an idea to create a GUC parameter for it.
Also maybe create a GUC parameter for the number of occurring WARNINGS by
rows with errors.
For CIM_MULTI and CIM_MULTI_CONDITIONAL cases the buffer is not needed.
It is needed for the CIM_SINGLE case.
Tests:
-- CIM_MULTI case
CREATE TABLE check_ign_err (n int, m int, k int);
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
1 1 1
2 2 2 2
3 3
a 4 4
5 b b
7 7 7
\.
SELECT * FROM check_ign_err;
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
n | m | k
---+---+---
1 | 1 | 1
7 | 7 | 7
(2 rows)
##################################################
-- CIM_SINGLE case
-- BEFORE row trigger
CREATE TABLE trig_test(n int, m int);
CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
BEGIN
INSERT INTO trig_test VALUES(NEW.n, NEW.m);
RETURN NEW;
END;
' LANGUAGE plpgsql;
CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
1 1 1
2 2 2 2
3 3
a 4 4
5 b b
7 7 7
\.
SELECT * FROM check_ign_err;
n | m | k
---+---+---
1 | 1 | 1
7 | 7 | 7
(2 rows)
##################################################
-- INSTEAD OF row trigger
CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
BEGIN
INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
RETURN NEW;
END;
' LANGUAGE plpgsql;
CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
SELECT * FROM check_ign_err;
1 1 1
2 2 2 2
3 3
a 4 4
5 b b
7 7 7
\.
SELECT * FROM check_ign_err_view;
n | m | k
---+---+---
1 | 1 | 1
7 | 7 | 7
(2 rows)
##################################################
-- foreign table case in postgres_fdw extension
##################################################
-- volatile function in WHERE clause
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
SELECT * FROM check_ign_err;
1 1 1
2 2 2 2
3 3
a 4 4
5 b b
7 7 7
\.
SELECT * FROM check_ign_err;
n | m | k
---+---+---
1 | 1 | 1
(1 row)
##################################################
-- CIM_MULTI_CONDITIONAL case
-- INSERT triggers for partition tables
CREATE TABLE check_ign_err (n int, m int, k int) PARTITION BY RANGE (n);
CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
FOR VALUES FROM (1) TO (4);
CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
FOR VALUES FROM (4) TO (8);
CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
BEGIN
INSERT INTO trig_test VALUES(NEW.n, NEW.m);
RETURN NEW;
END;
' LANGUAGE plpgsql;
CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
WARNING: COPY check_ign_err, line 2: "2 2 2 2"
WARNING: COPY check_ign_err, line 3: "3 3"
WARNING: COPY check_ign_err, line 4, column n: "a"
WARNING: COPY check_ign_err, line 5, column m: "b"
WARNING: COPY check_ign_err, line 6, column n: ""
SELECT * FROM check_ign_err;
1 1 1
2 2 2 2
3 3
a 4 4
5 b b
7 7 7
\.
n | m | k
---+---+---
1 | 1 | 1
7 | 7 | 7
(2 rows)
Thanks for feedback.
Regards, Damir
Attachments:
[application/x-patch] 0002-COPY-IGNORE_ERRORS.patch (17.3K, ../../CALH1LgvmNcnO8dYyckcEmiJ6PGDnpRWA3V1td1SDKzqt6FrMJw@mail.gmail.com/3-0002-COPY-IGNORE_ERRORS.patch)
download | inline diff:
From 6bf2168cd962b3cce666a2cabf082f558eec848c Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Oct 2021 11:55:18 +0300
Subject: [PATCH] COPY IGNORE_ERRORS
---
doc/src/sgml/ref/copy.sgml | 13 +++
src/backend/commands/copy.c | 8 ++
src/backend/commands/copyfrom.c | 138 +++++++++++++++++++++++++++-
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 118 ++++++++++++++++++++++++
src/test/regress/sql/copy2.sql | 110 ++++++++++++++++++++++
9 files changed, 395 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 8aae711b3b..7d20b1649e 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,18 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drop rows that contain malformed data while copying. That is rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows that result in constraint violations, rows containing columns where
+ the data type's input function raises an error.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3ac731803b..fead1aba46 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -402,6 +402,7 @@ ProcessCopyOptions(ParseState *pstate,
{
bool format_specified = false;
bool freeze_specified = false;
+ bool ignore_errors_specified = false;
bool header_specified = false;
ListCell *option;
@@ -442,6 +443,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index a976008b3d..b994697b9d 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -535,6 +535,7 @@ CopyFrom(CopyFromState cstate)
ExprContext *econtext;
TupleTableSlot *singleslot = NULL;
MemoryContext oldcontext = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
PartitionTupleRouting *proute = NULL;
ErrorContextCallback errcallback;
@@ -549,6 +550,17 @@ CopyFrom(CopyFromState cstate)
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ /* variables for copy from ignore_errors option */
+#define REPLAY_BUFFER_SIZE 3
+ HeapTuple replay_buffer[REPLAY_BUFFER_SIZE];
+ HeapTuple replay_tuple;
+ int saved_tuples = 0;
+ int replayed_tuples = 0;
+ bool replay_is_active = false;
+ bool begin_subtransaction = true;
+ bool find_error = false;
+ bool last_replaying = false;
+
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
@@ -855,9 +867,129 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ /*
+ * If option IGNORE_ERRORS is enabled, COPY skip rows with errors.
+ * NextCopyFrom() directly store the values/nulls array in the slot.
+ */
+ if (cstate->opts.ignore_errors)
+ {
+ bool valid_row = true;
+ bool skip_row = false;
+
+ PG_TRY();
+ {
+ if (!replay_is_active)
+ {
+ if (begin_subtransaction)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = oldowner;
+ }
+
+ if (saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+ if (valid_row)
+ {
+ if (insertMethod == CIM_SINGLE)
+ {
+ MemoryContextSwitchTo(oldcontext);
+
+ replay_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+ replay_buffer[saved_tuples++] = replay_tuple;
+
+ if (find_error)
+ skip_row = true;
+ }
+
+ begin_subtransaction = false;
+ }
+ }
+ else
+ {
+ ReleaseCurrentSubTransaction();
+
+ replay_is_active = true;
+ begin_subtransaction = true;
+ skip_row = true;
+ }
+ }
+ else
+ {
+ if (insertMethod == CIM_SINGLE && find_error && replayed_tuples < saved_tuples)
+ {
+ heap_deform_tuple(replay_buffer[replayed_tuples], RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+ replayed_tuples++;
+ }
+ else
+ {
+ MemSet(replay_buffer, 0, REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ saved_tuples = 0;
+ replayed_tuples = 0;
+
+ replay_is_active = false;
+ find_error = false;
+ skip_row = true;
+ }
+ }
+ }
+ PG_CATCH();
+ {
+ ErrorData *errdata;
+ MemoryContextSwitchTo(oldcontext);
+ errdata = CopyErrorData();
+
+ switch (errdata->sqlerrcode)
+ {
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ RollbackAndReleaseCurrentSubTransaction();
+ elog(WARNING, "%s", errdata->context);
+
+ begin_subtransaction = true;
+ find_error = true;
+ skip_row = true;
+
+ break;
+
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+ }
+ PG_END_TRY();
+
+ if (!valid_row)
+ {
+ if (!last_replaying)
+ {
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = oldowner;
+
+ if (replayed_tuples < saved_tuples)
+ {
+ replay_is_active = true;
+ skip_row = true;
+ last_replaying = true;
+ }
+ else
+ break;
+ }
+ else
+ break;
+ }
+
+ if (skip_row)
+ continue;
+ }
+ else
+ {
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ }
ExecStoreVirtualTuple(myslot);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index df5ceea910..3bb7235b34 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -800,7 +800,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3456,6 +3456,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeInteger(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17814,6 +17818,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18393,6 +18398,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index e572f585ef..feaf18b043 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2742,7 +2742,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..2b696f99bc 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..74827ecca0 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,124 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+-- CIM_SINGLE case
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err_view, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 3"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column m: "b"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+SELECT * FROM check_ign_err_view;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int, k int) PARTITION BY RANGE (n);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..7eee78bccd 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,116 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE case
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err_view;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int, k int) PARTITION BY RANGE (n);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
--
2.25.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-08-15 12:29 ` torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2022-08-15 12:29 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers
On 2022-07-19 21:40, Damir Belyalov wrote:
> Hi!
>
> Improved my patch by adding block subtransactions.
> The block size is determined by the REPLAY_BUFFER_SIZE parameter.
> I used the idea of a buffer for accumulating tuples in it.
> If we read REPLAY_BUFFER_SIZE rows without errors, the subtransaction
> will be committed.
> If we find an error, the subtransaction will rollback and the buffer
> will be replayed containing tuples.
Thanks for working on this!
I tested 0002-COPY-IGNORE_ERRORS.patch and faced an unexpected behavior.
I loaded 10000 rows which contained 1 wrong row.
I expected I could see 9999 rows after COPY, but just saw 999 rows.
Since when I changed MAX_BUFFERED_TUPLES from 1000 to other values, the
number of loaded rows also changed, I imagine MAX_BUFFERED_TUPLES might
be giving influence of this behavior.
```sh
$ cat /tmp/test10000.dat
1 aaa
2 aaa
3 aaa
4 aaa
5 aaa
6 aaa
7 aaa
8 aaa
9 aaa
10 aaa
11 aaa
...
9994 aaa
9995 aaa
9996 aaa
9997 aaa
9998 aaa
9999 aaa
xxx aaa
```
```SQL
=# CREATE TABLE test (id int, data text);
=# COPY test FROM '/tmp/test10000.dat' WITH (IGNORE_ERRORS);
WARNING: COPY test, line 10000, column i: "xxx"
COPY 9999
=# SELECT COUNT(*) FROM test;
count
-------
999
(1 row)
```
BTW I may be overlooking it, but have you submit this proposal to the
next CommitFest?
https://commitfest.postgresql.org/39/
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2022-08-15 13:23 ` Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-08-15 13:23 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; [email protected]
>
>
> Thank you for feedback.
I improved my patch recently and tested it on different sizes of
MAX_BUFFERED_TUPLES and REPLAY_BUFFER_SIZE.
> I loaded 10000 rows which contained 1 wrong row.
> I expected I could see 9999 rows after COPY, but just saw 999 rows.
Also I implemented your case and it worked correctly.
> BTW I may be overlooking it, but have you submit this proposal to the
next CommitFest?
Good idea. Haven't done it yet.
Regards,
Damir
Postgres Professional
Attachments:
[text/x-patch] 0003-COPY_IGNORE_ERRORS.patch (20.8K, ../../CALH1Lgu0dqsQUm9QYdoSvgu-O9Ac5OsbE3TxxZqChUrVF+Qrjw@mail.gmail.com/3-0003-COPY_IGNORE_ERRORS.patch)
download | inline diff:
From fa6b99c129eb890b25f006bb7891a247c8a431a7 Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Oct 2021 11:55:18 +0300
Subject: [PATCH] COPY_IGNORE_ERRORS without GUC with function
safeNextCopyFrom() with struct SafeCopyFromState with refactoring
---
doc/src/sgml/ref/copy.sgml | 13 ++
src/backend/commands/copy.c | 8 ++
src/backend/commands/copyfrom.c | 162 ++++++++++++++++++++++-
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 21 +++
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 123 +++++++++++++++++
src/test/regress/sql/copy2.sql | 110 +++++++++++++++
10 files changed, 445 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 8aae711b3b..7d20b1649e 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,18 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drop rows that contain malformed data while copying. That is rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows that result in constraint violations, rows containing columns where
+ the data type's input function raises an error.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3ac731803b..fead1aba46 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -402,6 +402,7 @@ ProcessCopyOptions(ParseState *pstate,
{
bool format_specified = false;
bool freeze_specified = false;
+ bool ignore_errors_specified = false;
bool header_specified = false;
ListCell *option;
@@ -442,6 +443,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index a976008b3d..285c491ddd 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -106,6 +106,9 @@ static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static bool safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext,
+ Datum *values, bool *nulls);
+
/*
* error context callback for COPY FROM
*
@@ -521,6 +524,125 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Analog of NextCopyFrom() but ignore rows with errors while copying.
+ */
+static bool
+safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls)
+{
+ SafeCopyFromState *safecstate = cstate->safecstate;
+ bool valid_row = true;
+
+ safecstate->skip_row = false;
+
+ PG_TRY();
+ {
+ if (!safecstate->replay_is_active)
+ {
+ if (safecstate->begin_subtransaction)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = safecstate->oldowner;
+
+ safecstate->begin_subtransaction = false;
+ }
+
+ if (safecstate->saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ valid_row = NextCopyFrom(cstate, econtext, values, nulls);
+ if (valid_row)
+ {
+ /* Fill replay_buffer in oldcontext*/
+ MemoryContextSwitchTo(safecstate->oldcontext);
+ safecstate->replay_buffer[safecstate->saved_tuples++] = heap_form_tuple(RelationGetDescr(cstate->rel), values, nulls);
+
+ safecstate->skip_row = true;
+ }
+ else if (!safecstate->processed_remaining_tuples)
+ {
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+ if (safecstate->replayed_tuples < safecstate->saved_tuples)
+ {
+ /* Prepare to replay remaining tuples if they exist */
+ safecstate->replay_is_active = true;
+ safecstate->processed_remaining_tuples = true;
+ safecstate->skip_row = true;
+ return true;
+ }
+ }
+ }
+ else
+ {
+ /* Buffer was filled, commit subtransaction and prepare to replay */
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ safecstate->replay_is_active = true;
+ safecstate->begin_subtransaction = true;
+ safecstate->skip_row = true;
+ }
+ }
+ else
+ {
+ if (safecstate->replayed_tuples < safecstate->saved_tuples)
+ {
+ /* Replaying tuple */
+ heap_deform_tuple(safecstate->replay_buffer[safecstate->replayed_tuples++], RelationGetDescr(cstate->rel), values, nulls);
+ }
+ else
+ {
+ /* Clean up replay_buffer */
+ MemSet(safecstate->replay_buffer, 0, REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ safecstate->saved_tuples = safecstate->replayed_tuples = 0;
+
+ safecstate->replay_is_active = false;
+ safecstate->skip_row = true;
+ }
+ }
+ }
+ PG_CATCH();
+ {
+ ErrorData *errdata;
+ MemoryContextSwitchTo(safecstate->oldcontext);
+ errdata = CopyErrorData();
+
+ switch (errdata->sqlerrcode)
+ {
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ safecstate->errors++;
+ if (safecstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+
+ safecstate->begin_subtransaction = true;
+ safecstate->skip_row = true;
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+ }
+ PG_END_TRY();
+
+ if (!valid_row)
+ {
+ ereport(WARNING,
+ errmsg("FIND %d ERRORS", safecstate->errors));
+ return false;
+ }
+
+ return true;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -535,6 +657,7 @@ CopyFrom(CopyFromState cstate)
ExprContext *econtext;
TupleTableSlot *singleslot = NULL;
MemoryContext oldcontext = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
PartitionTupleRouting *proute = NULL;
ErrorContextCallback errcallback;
@@ -819,6 +942,23 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option*/
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->safecstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->safecstate->saved_tuples = 0;
+ cstate->safecstate->replayed_tuples = 0;
+ cstate->safecstate->errors = 0;
+ cstate->safecstate->replay_is_active = false;
+ cstate->safecstate->begin_subtransaction = true;
+ cstate->safecstate->processed_remaining_tuples = false;
+
+ cstate->safecstate->oldowner = oldowner;
+ cstate->safecstate->oldcontext = oldcontext;
+ cstate->safecstate->insertMethod = insertMethod;
+ }
+
for (;;)
{
TupleTableSlot *myslot;
@@ -855,9 +995,25 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ /*
+ * If option IGNORE_ERRORS is enabled, COPY skips rows with errors.
+ * NextCopyFrom() directly store the values/nulls array in the slot.
+ */
+ if (cstate->safecstate)
+ {
+ bool valid_row = safeNextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+
+ /* Cannot continue or break in PG_TRY in safeNextCopyFrom() */
+ if (cstate->safecstate->skip_row)
+ continue;
+ if (!valid_row)
+ break;
+ }
+ else
+ {
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ }
ExecStoreVirtualTuple(myslot);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index df5ceea910..3bb7235b34 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -800,7 +800,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3456,6 +3456,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeInteger(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17814,6 +17818,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18393,6 +18398,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index e572f585ef..feaf18b043 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2742,7 +2742,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..2b696f99bc 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 3df1c5a97c..d9d3af1fb4 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,8 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
+
/*
* Represents the different source cases we need to worry about at
@@ -49,6 +51,24 @@ typedef enum CopyInsertMethod
CIM_MULTI_CONDITIONAL /* use table_multi_insert only if valid */
} CopyInsertMethod;
+/* Struct that holding fields for ignore_errors option. */
+typedef struct SafeCopyFromState
+{
+#define REPLAY_BUFFER_SIZE 1000
+ HeapTuple replay_buffer[REPLAY_BUFFER_SIZE]; /* accumulates tuples for replaying it after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active;
+ bool begin_subtransaction;
+ bool processed_remaining_tuples; /* for case of replaying last tuples */
+ bool skip_row;
+
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+ CopyInsertMethod insertMethod;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -71,6 +91,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *safecstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..ab1f059a02 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,129 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: FIND 5 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+-- CIM_SINGLE case
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: FIND 5 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err_view, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 3"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column m: "b"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: FIND 5 ERRORS
+SELECT * FROM check_ign_err_view;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: FIND 5 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int, k int) PARTITION BY RANGE (n);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 2 2 2"
+WARNING: COPY check_ign_err, line 3: "3 3"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column m: "b"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: FIND 5 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+---+---
+ 1 | 1 | 1
+ 7 | 7 | 7
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..7eee78bccd 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,116 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int, k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE case
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err_view;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int, k int) PARTITION BY RANGE (n);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 1 1
+2 2 2 2
+3 3
+a 4 4
+5 b b
+
+7 7 7
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
--
2.25.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-08-22 12:46 ` torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2022-08-22 12:46 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 2022-08-15 22:23, Damir Belyalov wrote:
>> I expected I could see 9999 rows after COPY, but just saw 999 rows.
> Also I implemented your case and it worked correctly.
Thanks for the new patch!
Here are some comments on it.
> + if (safecstate->saved_tuples < REPLAY_BUFFER_SIZE)
> + {
> + valid_row = NextCopyFrom(cstate, econtext, values,
> nulls);
> + if (valid_row)
> + {
> + /* Fill replay_buffer in oldcontext*/
> + MemoryContextSwitchTo(safecstate->oldcontext);
> +
> safecstate->replay_buffer[safecstate->saved_tuples++] =
> heap_form_tuple(RelationGetDescr(cstate->rel), values, nulls);
>
> + /* Buffer was filled, commit subtransaction and prepare
> to replay */
> + ReleaseCurrentSubTransaction();
What is actually being committed by this ReleaseCurrentSubTransaction()?
It seems to me that just safecstate->replay_buffer is fulfilled before
this commit.
As a test, I rewrote this ReleaseCurrentSubTransaction() to
RollbackAndReleaseCurrentSubTransaction() and COPYed over 1000 rows of
data, but same data were loaded.
> +#define REPLAY_BUFFER_SIZE 1000
I feel it might be better to have it as a parameter rather than fixed at
1000.
> +/*
> + * Analog of NextCopyFrom() but ignore rows with errors while copying.
> + */
> +static bool
> +safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext, Datum
> *values, bool *nulls)
NextCopyFrom() is in copyfromparse.c while safeNextCopyFrom() is in
copyfrom.c.
Since safeNextCopyFrom() is analog of NextCopyFrom() as commented, would
it be natural to put them in the same file?
> 188 + safecstate->errors++;
> 189 + if (safecstate->errors <= 100)
> 190 + ereport(WARNING,
> 191 + (errcode(errdata->sqlerrcode),
> 192 + errmsg("%s", errdata->context)));
It would be better to state in the manual that errors exceeding 100 are
not displayed.
Or, it might be acceptable to output all errors that exceed 100.
> +typedef struct SafeCopyFromState
> +{
> +#define REPLAY_BUFFER_SIZE 1000
> + HeapTuple replay_buffer[REPLAY_BUFFER_SIZE]; /* accumulates
> tuples for replaying it after an error */
> + int saved_tuples; /* # of tuples in
> replay_buffer */
> + int replayed_tuples; /* # of tuples was
> replayed from buffer */
> + int errors; /* total # of errors */
> + bool replay_is_active;
> + bool begin_subtransaction;
> + bool processed_remaining_tuples; /* for case of
> replaying last tuples */
> + bool skip_row;
It would be helpful to add comments about skip_row, etc.
```
$ git apply ../patch/0003-COPY_IGNORE_ERRORS.patch
../patch/0003-COPY_IGNORE_ERRORS.patch:86: indent with spaces.
Datum *values, bool *nulls);
warning: 1 line adds whitespace errors.
```
There was a warning when applying the patch.
```
=# copy test from '/tmp/10000.data' with (ignore_errors);
WARNING: FIND 0 ERRORS
COPY 1003
```
When there were no errors, this WARNING seems not necessary.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2022-08-24 16:54 ` Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-08-24 16:54 UTC (permalink / raw)
To: torikoshia <[email protected]>; pgsql-hackers; [email protected]
>
> > + /* Buffer was filled, commit subtransaction and prepare
> to replay */
> > + ReleaseCurrentSubTransaction();
> What is actually being committed by this ReleaseCurrentSubTransaction()?
> It seems to me that just safecstate->replay_buffer is fulfilled before
> this commit.
>
All tuples are collected in replay_buffer, which is created in
''oldcontext'' of CopyFrom() (not context of a subtransaction). That's why
it didn't clean up when you used RollbackAndReleaseCurrentSubTransaction().
Subtransactions are needed for better safety. There is no error when
copying from a file to the replay_buffer, but an error may occur at the
next stage - when adding a tuple to the table. Also there may be other
errors that are not obvious at first glance.
I feel it might be better to have it as a parameter rather than fixed at
> 1000.
>
Yes, I thought about it too. So I created another version with the GUC
parameter - replay_buffer_size. Attached it. But I think users won't need
to change replay_buffer_size.
Also replay_buffer does the same thing as MultiInsert buffer does and
MultiInsert buffer defined by const = 1000.
NextCopyFrom() is in copyfromparse.c while safeNextCopyFrom() is in
> copyfrom.c.
> Since safeNextCopyFrom() is analog of NextCopyFrom() as commented, would
> it be natural to put them in the same file?
>
Sure, corrected it.
> > 188 + safecstate->errors++;
> > 189 + if (safecstate->errors <= 100)
> > 190 + ereport(WARNING,
> > 191 + (errcode(errdata->sqlerrcode),
> > 192 + errmsg("%s", errdata->context)));
>
> It would be better to state in the manual that errors exceeding 100 are
> not displayed.
> Or, it might be acceptable to output all errors that exceed 100.
>
It'll be too complicated to create a new parameter just for showing the
given number of errors. I think 100 is an optimal size.
> > +typedef struct SafeCopyFromState
> > +{
> > +#define REPLAY_BUFFER_SIZE 1000
> > + HeapTuple replay_buffer[REPLAY_BUFFER_SIZE]; /* accumulates
> > tuples for replaying it after an error */
> > + int saved_tuples; /* # of tuples in
> > replay_buffer */
> > + int replayed_tuples; /* # of tuples was
> > replayed from buffer */
> > + int errors; /* total # of errors */
> > + bool replay_is_active;
> > + bool begin_subtransaction;
> > + bool processed_remaining_tuples; /* for case of
> > replaying last tuples */
> > + bool skip_row;
>
> It would be helpful to add comments about skip_row, etc.
>
Corrected it.
> WARNING: FIND 0 ERRORS
> When there were no errors, this WARNING seems not necessary.
>
Corrected it.
Add to this patch processing other errors and constraints and tests for
them.
I had to create another function safeExecConstraints() only for processing
constraints, because ExecConstraints() is after NextCopyFrom() and is not
in PG_TRY. This thing a little bit complicated the code.
Maybe it is a good approach to create a new function SafeCopyFrom() and do
all ''safe copying'' in PG_TRY, but it will almost duplicate the CopyFrom()
code.
Or maybe create a function only for loop for(;;). But we have the same
thing with duplicating code and passing a lot of variables (which are
created at the beginning of CopyFrom()) to this function.
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-08-24 16:57 ` Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-08-24 16:57 UTC (permalink / raw)
To: torikoshia <[email protected]>; pgsql-hackers; [email protected]
>
Attachments:
[text/x-patch] 0004-COPY_IGNORE_ERRORS.patch (30.6K, ../../CALH1LgvzpORM+EHVXRz4-Dv+_aHvm9m9Hu6eYM4PfppwKgXvFQ@mail.gmail.com/3-0004-COPY_IGNORE_ERRORS.patch)
download | inline diff:
From 09befdad45a6b1ae70d6c5abc90d1c2296e56ee1 Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Oct 2021 11:55:18 +0300
Subject: [PATCH] COPY_IGNORE_ERRORS with GUC for replay_buffer size
---
doc/src/sgml/config.sgml | 17 ++
doc/src/sgml/ref/copy.sgml | 19 ++
src/backend/commands/copy.c | 8 +
src/backend/commands/copyfrom.c | 114 +++++++++++-
src/backend/commands/copyfromparse.c | 169 ++++++++++++++++++
src/backend/parser/gram.y | 8 +-
src/backend/utils/misc/guc.c | 11 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 6 +
src/include/commands/copyfrom_internal.h | 19 ++
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 130 ++++++++++++++
src/test/regress/sql/copy2.sql | 116 ++++++++++++
14 files changed, 617 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 37fd80388c..69373b8d8c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1961,6 +1961,23 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="replay_buffer_size">
+ <term><varname>replay_buffer_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replay_buffer_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the size of buffer for COPY FROM IGNORE_ERRORS option. This buffer
+ is created when subtransaction begins and accumulates tuples until an error
+ occurs. Then it starts replaying stored tuples. the buffer size is the size
+ of the subtransaction. Therefore, on large tables, in order to avoid the
+ error of the maximum number of subtransactions, it should be increased.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 8aae711b3b..7ff6f6dea7 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,24 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drop rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows that result in constraint violations, rows containing columns where
+ the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ The option is implemented with subtransactions and a buffer created in
+ them, which accumulates tuples until an error occures.
+ The size of buffer is the size of subtransaction block.
+ It is a GUC parameter "replay_buffer_size" and equals 1000 by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3ac731803b..fead1aba46 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -402,6 +402,7 @@ ProcessCopyOptions(ParseState *pstate,
{
bool format_specified = false;
bool freeze_specified = false;
+ bool ignore_errors_specified = false;
bool header_specified = false;
ListCell *option;
@@ -442,6 +443,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index a976008b3d..7e997d15c6 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -73,6 +73,11 @@
/* Trim the list of buffers back down to this number after flushing */
#define MAX_PARTITION_BUFFERS 32
+/*
+ * GUC parameters
+ */
+int replay_buffer_size;
+
/* Stores multi-insert data related to a single relation in CopyFrom. */
typedef struct CopyMultiInsertBuffer
{
@@ -100,12 +105,13 @@ typedef struct CopyMultiInsertInfo
int ti_options; /* table insert options */
} CopyMultiInsertInfo;
-
/* non-export function prototypes */
static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static void safeExecConstraints(CopyFromState cstate, ResultRelInfo *resultRelInfo, TupleTableSlot *myslot, EState *estate);
+
/*
* error context callback for COPY FROM
*
@@ -521,6 +527,61 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Ignore constraints if IGNORE_ERRORS is enabled
+ */
+static void
+safeExecConstraints(CopyFromState cstate, ResultRelInfo *resultRelInfo, TupleTableSlot *myslot, EState *estate)
+{
+ SafeCopyFromState *safecstate = cstate->safecstate;
+
+ safecstate->skip_row = false;
+
+ PG_TRY();
+ ExecConstraints(resultRelInfo, myslot, estate);
+ PG_CATCH();
+ {
+ ErrorData *errdata;
+ MemoryContext cxt;
+
+ cxt = MemoryContextSwitchTo(safecstate->oldcontext);
+ errdata = CopyErrorData();
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore Constraint Violation */
+ case ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION:
+ case ERRCODE_RESTRICT_VIOLATION:
+ case ERRCODE_NOT_NULL_VIOLATION:
+ case ERRCODE_FOREIGN_KEY_VIOLATION:
+ case ERRCODE_UNIQUE_VIOLATION:
+ case ERRCODE_CHECK_VIOLATION:
+ case ERRCODE_EXCLUSION_VIOLATION:
+ safecstate->errors++;
+ if (cstate->safecstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+
+ safecstate->begin_subtransaction = true;
+ safecstate->skip_row = true;
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(cxt);
+ }
+ PG_END_TRY();
+}
+
/*
* Copy FROM file to relation.
*/
@@ -535,6 +596,7 @@ CopyFrom(CopyFromState cstate)
ExprContext *econtext;
TupleTableSlot *singleslot = NULL;
MemoryContext oldcontext = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
PartitionTupleRouting *proute = NULL;
ErrorContextCallback errcallback;
@@ -819,9 +881,30 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option */
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->safecstate = palloc(sizeof(SafeCopyFromState));
+
+ /* Create replay_buffer in oldcontext */
+ cstate->safecstate->replay_buffer = (HeapTuple *) palloc(replay_buffer_size * sizeof(HeapTuple));
+
+ cstate->safecstate->saved_tuples = 0;
+ cstate->safecstate->replayed_tuples = 0;
+ cstate->safecstate->errors = 0;
+ cstate->safecstate->replay_is_active = false;
+ cstate->safecstate->begin_subtransaction = true;
+ cstate->safecstate->processed_remaining_tuples = false;
+
+ cstate->safecstate->oldowner = oldowner;
+ cstate->safecstate->oldcontext = oldcontext;
+ cstate->safecstate->insertMethod = insertMethod;
+ }
+
for (;;)
{
TupleTableSlot *myslot;
+ bool valid_row;
bool skip_tuple;
CHECK_FOR_INTERRUPTS();
@@ -855,8 +938,21 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ /*
+ * NextCopyFrom() directly store the values/nulls array in the slot.
+ * safeNextCopyFrom() ignores rows with errors if IGNORE_ERRORS is enabled.
+ */
+ if (!cstate->safecstate)
+ valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+ else
+ {
+ valid_row = safeNextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+
+ if (cstate->safecstate->skip_row)
+ continue;
+ }
+
+ if (!valid_row)
break;
ExecStoreVirtualTuple(myslot);
@@ -1035,7 +1131,17 @@ CopyFrom(CopyFromState cstate)
*/
if (resultRelInfo->ri_FdwRoutine == NULL &&
resultRelInfo->ri_RelationDesc->rd_att->constr)
- ExecConstraints(resultRelInfo, myslot, estate);
+ {
+ if (cstate->opts.ignore_errors)
+ {
+ safeExecConstraints(cstate, resultRelInfo, myslot, estate);
+
+ if (cstate->safecstate->skip_row)
+ continue;
+ }
+ else
+ ExecConstraints(resultRelInfo, myslot, estate);
+ }
/*
* Also check the tuple against the partition constraint, if
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 57813b3458..1aae27d80d 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -1026,6 +1026,175 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
return true;
}
+/*
+ * Analog of NextCopyFrom() but skips rows with errors while copying.
+ */
+bool
+safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls)
+{
+ SafeCopyFromState *safecstate = cstate->safecstate;
+ bool valid_row = true;
+
+ safecstate->skip_row = false;
+
+ PG_TRY();
+ {
+ if (!safecstate->replay_is_active)
+ {
+ if (safecstate->begin_subtransaction)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = safecstate->oldowner;
+
+ safecstate->begin_subtransaction = false;
+ }
+
+ if (safecstate->saved_tuples < replay_buffer_size)
+ {
+ valid_row = NextCopyFrom(cstate, econtext, values, nulls);
+ if (valid_row)
+ {
+ /* Fill replay_buffer in CopyFrom() oldcontext */
+ MemoryContext cxt = MemoryContextSwitchTo(safecstate->oldcontext);
+
+ safecstate->replay_buffer[safecstate->saved_tuples++] = heap_form_tuple(RelationGetDescr(cstate->rel), values, nulls);
+ MemoryContextSwitchTo(cxt);
+
+ safecstate->skip_row = true;
+ }
+ else if (!safecstate->processed_remaining_tuples)
+ {
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ if (safecstate->replayed_tuples < safecstate->saved_tuples)
+ {
+ /* Prepare to replay remaining tuples if they exist */
+ safecstate->replay_is_active = true;
+ safecstate->processed_remaining_tuples = true;
+ safecstate->skip_row = true;
+ return true;
+ }
+ }
+ }
+ else
+ {
+ /* Buffer was filled, commit subtransaction and prepare to replay */
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ safecstate->replay_is_active = true;
+ safecstate->begin_subtransaction = true;
+ safecstate->skip_row = true;
+ }
+ }
+ else
+ {
+ if (safecstate->replayed_tuples < safecstate->saved_tuples)
+ /* Replaying tuple */
+ heap_deform_tuple(safecstate->replay_buffer[safecstate->replayed_tuples++], RelationGetDescr(cstate->rel), values, nulls);
+ else
+ {
+ /* Clean up replay_buffer */
+ MemSet(safecstate->replay_buffer, 0, replay_buffer_size * sizeof(HeapTuple));
+ safecstate->saved_tuples = safecstate->replayed_tuples = 0;
+
+ safecstate->replay_is_active = false;
+ safecstate->skip_row = true;
+ }
+ }
+ }
+ PG_CATCH();
+ {
+ ErrorData *errdata;
+ MemoryContext cxt;
+
+ cxt = MemoryContextSwitchTo(safecstate->oldcontext);
+ errdata = CopyErrorData();
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = safecstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore malformed data */
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ safecstate->errors++;
+ if (safecstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+
+ safecstate->begin_subtransaction = true;
+ safecstate->skip_row = true;
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(cxt);
+ }
+ PG_END_TRY();
+
+ if (!valid_row)
+ {
+ if (safecstate->errors == 0)
+ ereport(NOTICE,
+ errmsg("FIND %d ERRORS", safecstate->errors));
+ else if (safecstate->errors == 1)
+ ereport(WARNING,
+ errmsg("FIND %d ERROR", safecstate->errors));
+ else
+ ereport(WARNING,
+ errmsg("FIND %d ERRORS", safecstate->errors));
+
+ return false;
+ }
+
+ return true;
+}
+
/*
* Read the next input line and stash it in line_buf.
*
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index df5ceea910..3bb7235b34 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -800,7 +800,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3456,6 +3456,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeInteger(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17814,6 +17818,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18393,6 +18398,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 0328029d43..54209a4a3c 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -49,6 +49,7 @@
#include "catalog/pg_parameter_acl.h"
#include "catalog/storage.h"
#include "commands/async.h"
+#include "commands/copy.h"
#include "commands/prepare.h"
#include "commands/tablespace.h"
#include "commands/trigger.h"
@@ -2527,6 +2528,16 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replay_buffer_size", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the size of replay buffer for COPY FROM IGNORE_ERRORS option"),
+ NULL
+ },
+ &replay_buffer_size,
+ 1000, 1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index b4bc06e5f5..f4e777a0a3 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -155,6 +155,8 @@
# mmap
# (change requires restart)
#min_dynamic_shared_memory = 0MB # (change requires restart)
+#replay_buffer_size = 1000 # min 1
+ # (change requires restart)
# - Disk -
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index e572f585ef..feaf18b043 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2742,7 +2742,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..fc9f559efe 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -19,6 +19,9 @@
#include "parser/parse_node.h"
#include "tcop/dest.h"
+/* User-settable GUC parameters */
+extern PGDLLIMPORT int replay_buffer_size;
+
/*
* Represents whether a header line should be present, and whether it must
* match the actual names (which implies "true").
@@ -42,6 +45,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
@@ -78,6 +82,8 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Datum *values, bool *nulls);
+extern bool safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext,
+ Datum *values, bool *nulls);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 3df1c5a97c..4227a7babd 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -49,6 +50,23 @@ typedef enum CopyInsertMethod
CIM_MULTI_CONDITIONAL /* use table_multi_insert only if valid */
} CopyInsertMethod;
+ /* Struct that holding fields for COPY FROM IGNORE_ERRORS option. */
+typedef struct SafeCopyFromState
+{
+ HeapTuple *replay_buffer; /* accumulates tuples for replaying them after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active; /* become active after an error */
+ bool begin_subtransaction; /* if it's true, we can start subtransaction */
+ bool processed_remaining_tuples; /* for case of replaying last tuples */
+ bool skip_row; /* if it's true, we should skip this row */
+
+ MemoryContext oldcontext; /* create repay_buffer in CopyFrom() context */
+ ResourceOwner oldowner; /* CopyFrom() resource owner */
+ CopyInsertMethod insertMethod;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -71,6 +89,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *safecstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..093c7958be 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,136 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int check (n < 8), m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column n: "5 {5} 5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: COPY check_ign_err, line 8, column n: "8 {8} 8"
+WARNING: FIND 7 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[]);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column n: "5 {5} 5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: COPY check_ign_err, line 8, column n: "8 {8} 8"
+WARNING: FIND 7 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column n: "5 {5} 5555555555"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: COPY check_ign_err_view, line 8, column n: "8 {8} 8"
+WARNING: FIND 7 ERRORS
+SELECT * FROM check_ign_err_view;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column n: "5 {5} 5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: COPY check_ign_err, line 8, column n: "8 {8} 8"
+WARNING: FIND 7 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int check (n < 8), m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column n: "5 {5} 5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: COPY check_ign_err, line 8, column n: "8 {8} 8"
+WARNING: FIND 7 ERRORS
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..c91122aa1e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int check (n < 8), m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[]);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO check_ign_err VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err_view;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* find values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int check (n < 8), m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (8);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
--
2.25.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-08-29 13:02 ` torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2022-08-29 13:02 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 2022-08-25 01:54, Damir Belyalov wrote:
>>> + /* Buffer was filled, commit subtransaction and
>> prepare to replay */
>>> + ReleaseCurrentSubTransaction();
>> What is actually being committed by this
>> ReleaseCurrentSubTransaction()?
>> It seems to me that just safecstate->replay_buffer is fulfilled
>> before
>> this commit.
>
> All tuples are collected in replay_buffer, which is created in
> ''oldcontext'' of CopyFrom() (not context of a subtransaction). That's
> why it didn't clean up when you used
> RollbackAndReleaseCurrentSubTransaction().
> Subtransactions are needed for better safety. There is no error when
> copying from a file to the replay_buffer, but an error may occur at
> the next stage - when adding a tuple to the table. Also there may be
> other errors that are not obvious at first glance.
Thanks for the explanation and updating patch.
I now understand that the data being COPYed are not the target of
COMMIT.
Although in past discussions[1] it seems the data to be COPYed were also
subject to COMMIT, but I understand this patch has adopted another
design.
350 + /* Buffer was filled, commit subtransaction and
prepare to replay */
351 + ReleaseCurrentSubTransaction();
352 + CurrentResourceOwner = safecstate->oldowner;
353 +
354 + safecstate->replay_is_active = true;
BTW in v4 patch, data are loaded into the buffer one by one, and when
the buffer fills up, the data in the buffer are 'replayed' also one by
one, right?
Wouldn't this have more overhead than a normal COPY?
As a test, I COPYed slightly larger data with and without ignore_errors
option.
There might be other reasons, but I found a performance difference.
```
=# copy test from '/tmp/10000000.data' ;
COPY 10000000
Time: 6001.325 ms (00:06.001)
=# copy test from '/tmp/10000000.data' with (ignore_errors);
NOTICE: FIND 0 ERRORS
COPY 10000000
Time: 7711.555 ms (00:07.712)
```
>> I feel it might be better to have it as a parameter rather than
>> fixed at
>> 1000.
>
> Yes, I thought about it too. So I created another version with the GUC
> parameter - replay_buffer_size. Attached it. But I think users won't
> need to change replay_buffer_size.
> Also replay_buffer does the same thing as MultiInsert buffer does and
> MultiInsert buffer defined by const = 1000.
Yeah, when the data being COPYed are not the target of COMMIT, I also
think users won't neet to change it.
>
>> NextCopyFrom() is in copyfromparse.c while safeNextCopyFrom() is in
>> copyfrom.c.
>> Since safeNextCopyFrom() is analog of NextCopyFrom() as commented,
>> would
>> it be natural to put them in the same file?
>
> Sure, corrected it.
Thanks.
>
>>> 188 + safecstate->errors++;
>>> 189 + if (safecstate->errors <= 100)
>>> 190 + ereport(WARNING,
>>> 191 + (errcode(errdata->sqlerrcode),
>>> 192 + errmsg("%s", errdata->context)));
>>
>> It would be better to state in the manual that errors exceeding 100
>> are
>> not displayed.
>> Or, it might be acceptable to output all errors that exceed 100.
>
> It'll be too complicated to create a new parameter just for showing
> the given number of errors. I think 100 is an optimal size.
Yeah, I may have misled you, but I also don't think this needs a new
parameter.
I just thought 'either' of the following would be better:
- Put in the documentation that the warnings will not be output for more
than 101 cases.
- Output all the warnings.
>> It would be helpful to add comments about skip_row, etc.
>
> Corrected it.
Thanks.
>
>> WARNING: FIND 0 ERRORS
>> When there were no errors, this WARNING seems not necessary.
>
> Corrected it.
Thanks.
I applied v4 patch and when canceled the COPY, there was a case I found
myself left in a transaction.
Should this situation be prevented from occurring?
```
=# copy test from '/tmp/10000000.data' with (ignore_errors );
^CCancel request sent
ERROR: canceling statement due to user request
=# truncate test;
ERROR: current transaction is aborted, commands ignored until end of
transaction block
```
[1]
https://www.postgresql.org/message-id/1197677930.1536.18.camel%40dell.linuxdev.us.dell.com
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2022-09-21 12:11 ` Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-09-21 12:11 UTC (permalink / raw)
To: torikoshia <[email protected]>; pgsql-hackers; [email protected]
Thank you for reviewing.
In the previous patch there was an error when processing constraints. The
patch was fixed, but the code grew up and became more complicated
(0005-COPY_IGNORE_ERRORS). I also simplified the logic of
safeNextCopyFrom().
You asked why we need subtransactions, so the answer is in this patch. When
processing a row that does not satisfy constraints or INSTEAD OF triggers,
it is necessary to rollback the subtransaction and return the table to its
original state.
Cause of complexity, I had to abandon the constraints, triggers processing
in and handle only errors that occur when reading the file. Attaching
simplified patch (0006-COPY_IGNORE_ERRORS).
Checked out these patches on all COPY regress tests and it worked correctly.
> BTW in v4 patch, data are loaded into the buffer one by one, and when
> the buffer fills up, the data in the buffer are 'replayed' also one by
> one, right?
> Wouldn't this have more overhead than a normal COPY?
>
The data is loaded into the buffer one by one, but in the "replay mode"
ignore_errors works as standard COPY. Tuples add to the slot and depending
on the type of the slot (CIM_SINGLE or CIM_MULTI) tuples are replayed in
the corresponding case.
For the 0006 patch you can imagine that we divide the loop for(;;) in 2
parts. The first part is adding tuples to the buffer and the second part is
inserting tuples to the table. These parts don't intersect with each other
and are completed sequentially.
The main idea of replay_buffer is that it is needed for the CIM_SINGLE
case. You can implement the CIM_SINGLE case and see that tuples before an
error occurring don't add to the table. Logic of the 0005 patch is similar
but with some differences.
As a test, I COPYed slightly larger data with and without ignore_errors
> option.
> There might be other reasons, but I found a performance difference.
Tried to reduce performance difference with cleaning up replay_buffer with
resetting the new context for replay_buffer - replay_cxt.
```
Before:
Without ignore_errors:
COPY 10000000
Time: 15538,579 ms (00:15,539)
With ignore_errors:
COPY 10000000
Time: 21289,121 ms (00:21,289)
After:
Without ignore_errors:
COPY 10000000
Time: 15318,922 ms (00:15,319)
With ignore_errors:
COPY 10000000
Time: 19868,175 ms (00:19,868)
```
- Put in the documentation that the warnings will not be output for more
> than 101 cases.
>
Yeah, I point it out in the doc.
> I applied v4 patch and when canceled the COPY, there was a case I found
> myself left in a transaction.
> Should this situation be prevented from occurring?
>
> ```
> =# copy test from '/tmp/10000000.data' with (ignore_errors );
>
> ^CCancel request sent
> ERROR: canceling statement due to user request
>
> =# truncate test;
> ERROR: current transaction is aborted, commands ignored until end of
> transaction block
> ```
>
Tried to implement your error and could not. The result was the same as
COPY FROM implements.
Attachments:
[text/x-patch] 0005-COPY_IGNORE_ERRORS.patch (27.1K, ../../CALH1Lgsr7Asz6r6jnRta7DVEjKwyz=b3XJhyCMoYv8Yy9y4cXg@mail.gmail.com/3-0005-COPY_IGNORE_ERRORS.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index c25b52d0cb..c99adabcc9 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,20 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows that result in constraint violations, rows containing columns where
+ the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 49924e476a..f41b25f26a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -406,6 +406,7 @@ ProcessCopyOptions(ParseState *pstate,
bool is_from,
List *options)
{
+ bool ignore_errors_specified = false;
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
@@ -448,6 +449,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index e8bb168aea..6474d47d29 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -106,6 +106,12 @@ static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static bool safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext,
+ Datum *values, bool *nulls);
+static bool safeExecConstraints(CopyFromState cstate, ResultRelInfo *resultRelInfo, TupleTableSlot *myslot, EState *estate);
+
+static void addToReplayBuffer(CopyFromState cstate, TupleTableSlot *myslot);
+
/*
* error context callback for COPY FROM
*
@@ -521,6 +527,254 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Analog of NextCopyFrom() but ignores rows with errors while copying.
+ */
+bool
+safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+
+ sfcstate->tuple_is_valid = false;
+
+ PG_TRY();
+ {
+ if (sfcstate->begin_subxact)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = false;
+ }
+
+ if (!sfcstate->replay_is_active)
+ {
+ if (sfcstate->saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ MemoryContext cxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+ bool valid_row = NextCopyFrom(cstate, econtext, values, nulls);
+
+ CurrentMemoryContext = cxt;
+
+ sfcstate->tuple_is_valid = true;
+
+ if (!valid_row && sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ /* Prepare for replaying remaining tuples if they exist */
+ sfcstate->replay_is_active = true;
+
+ /* If there are insteadof triggers we should rollback subtransaction */
+ if (sfcstate->has_instead_insert_row_trig)
+ {
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = true;
+ }
+ }
+ else if (!valid_row)
+ {
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ if (sfcstate->errors == 0)
+ ereport(NOTICE,
+ errmsg("%d errors", sfcstate->errors));
+ else if (sfcstate->errors == 1)
+ ereport(WARNING,
+ errmsg("%d error", sfcstate->errors));
+ else
+ ereport(WARNING,
+ errmsg("%d errors", sfcstate->errors));
+
+ return false;
+ }
+ }
+ else
+ {
+ /* Buffer was filled, prepare for replaying */
+ sfcstate->replay_is_active = true;
+
+ /* If there are insteadof triggers we should rollback subtransaction */
+ if (sfcstate->has_instead_insert_row_trig)
+ {
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = true;
+ }
+ }
+ }
+
+ if (sfcstate->replay_is_active)
+ {
+ if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ /* Replaying the tuple */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+
+ heap_deform_tuple(sfcstate->replay_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel), values, nulls);
+ MemoryContextSwitchTo(cxt);
+
+ sfcstate->tuple_is_valid = true;
+ }
+ else
+ {
+ /* Clean up replay_buffer */
+ MemoryContextReset(sfcstate->replay_cxt);
+ sfcstate->saved_tuples = sfcstate->replayed_tuples = 0;
+
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = true;
+ sfcstate->replay_is_active = false;
+ }
+ }
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore data exceptions */
+ case ERRCODE_CHARACTER_NOT_IN_REPERTOIRE:
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+
+ sfcstate->begin_subxact = true;
+
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ return true;
+}
+
+/*
+ * Analog of ExecConstraints(), but ignores rows in constraint violations.
+ */
+bool
+safeExecConstraints(CopyFromState cstate, ResultRelInfo *resultRelInfo, TupleTableSlot *myslot, EState *estate)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+
+ PG_TRY();
+ ExecConstraints(resultRelInfo, myslot, estate);
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore constraint violations */
+ case ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION:
+ case ERRCODE_RESTRICT_VIOLATION:
+ case ERRCODE_NOT_NULL_VIOLATION:
+ case ERRCODE_FOREIGN_KEY_VIOLATION:
+ case ERRCODE_UNIQUE_VIOLATION:
+ case ERRCODE_CHECK_VIOLATION:
+ case ERRCODE_EXCLUSION_VIOLATION:
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s %s", errdata->message, errdata->detail)));
+
+ sfcstate->begin_subxact = true;
+ sfcstate->tuple_is_valid = false;
+
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ if (!sfcstate->tuple_is_valid)
+ return false;
+
+ return true;
+}
+
+void
+addToReplayBuffer(CopyFromState cstate, TupleTableSlot *myslot)
+{
+ MemoryContext cxt = MemoryContextSwitchTo(cstate->sfcstate->replay_cxt);
+ HeapTuple saved_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+
+ cstate->sfcstate->replay_buffer[cstate->sfcstate->saved_tuples++] = saved_tuple;
+ MemoryContextSwitchTo(cxt);
+}
+
/*
* Copy FROM file to relation.
*/
@@ -535,6 +789,7 @@ CopyFrom(CopyFromState cstate)
ExprContext *econtext;
TupleTableSlot *singleslot = NULL;
MemoryContext oldcontext = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
PartitionTupleRouting *proute = NULL;
ErrorContextCallback errcallback;
@@ -819,6 +1074,27 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option */
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->sfcstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->sfcstate->replay_cxt = AllocSetContextCreate(oldcontext,
+ "Replay context",
+ ALLOCSET_DEFAULT_SIZES);
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ cstate->sfcstate->saved_tuples = 0;
+ cstate->sfcstate->replayed_tuples = 0;
+ cstate->sfcstate->errors = 0;
+ cstate->sfcstate->replay_is_active = false;
+ cstate->sfcstate->begin_subxact = true;
+ cstate->sfcstate->oldowner = oldowner;
+ cstate->sfcstate->oldcontext = oldcontext;
+ if (has_instead_insert_row_trig)
+ cstate->sfcstate->has_instead_insert_row_trig = true;
+ }
+
for (;;)
{
TupleTableSlot *myslot;
@@ -855,19 +1131,25 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ if (cstate->sfcstate)
+ {
+ /* If option IGNORE_ERRORS is enabled, COPY skips rows with errors */
+ if (!safeNextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ if (!cstate->sfcstate->tuple_is_valid)
+ continue;
+ }
+ else
+ {
+ /* Directly store the values/nulls array in the slot */
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ }
ExecStoreVirtualTuple(myslot);
- /*
- * Constraints and where clause might reference the tableoid column,
- * so (re-)initialize tts_tableOid before evaluating them.
- */
myslot->tts_tableOid = RelationGetRelid(target_resultRelInfo->ri_RelationDesc);
- /* Triggers and stuff need to be invoked in query context. */
MemoryContextSwitchTo(oldcontext);
if (cstate->whereClause)
@@ -1020,6 +1302,13 @@ CopyFrom(CopyFromState cstate)
if (has_instead_insert_row_trig)
{
ExecIRInsertTriggers(estate, resultRelInfo, myslot);
+
+ /* Add tuple to replay_buffer if IGNORE_ERRORS is enabled */
+ if (cstate->sfcstate && !cstate->sfcstate->replay_is_active)
+ {
+ addToReplayBuffer(cstate, myslot);
+ continue;
+ }
}
else
{
@@ -1035,7 +1324,15 @@ CopyFrom(CopyFromState cstate)
*/
if (resultRelInfo->ri_FdwRoutine == NULL &&
resultRelInfo->ri_RelationDesc->rd_att->constr)
- ExecConstraints(resultRelInfo, myslot, estate);
+ {
+ if (cstate->sfcstate)
+ {
+ if (!safeExecConstraints(cstate, resultRelInfo, myslot, estate))
+ continue;
+ }
+ else
+ ExecConstraints(resultRelInfo, myslot, estate);
+ }
/*
* Also check the tuple against the partition constraint, if
@@ -1047,6 +1344,13 @@ CopyFrom(CopyFromState cstate)
(proute == NULL || has_before_insert_row_trig))
ExecPartitionCheck(resultRelInfo, myslot, estate, true);
+ /* Add tuple to replay_buffer if IGNORE_ERRORS is enabled */
+ if (cstate->sfcstate && !cstate->sfcstate->replay_is_active)
+ {
+ addToReplayBuffer(cstate, myslot);
+ continue;
+ }
+
/* Store the slot in the multi-insert buffer, when enabled. */
if (insertMethod == CIM_MULTI || leafpart_use_multi_insert)
{
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b5ab9d9c9a..b49954c0aa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -808,7 +808,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3477,6 +3477,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeBoolean(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17778,6 +17782,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18357,6 +18362,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 62a39779b9..fe590ff7a8 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2748,7 +2748,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..2b696f99bc 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index e37c6032ae..5615fa55ef 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -49,6 +50,24 @@ typedef enum CopyInsertMethod
CIM_MULTI_CONDITIONAL /* use table_multi_insert only if valid */
} CopyInsertMethod;
+/* Struct that holding fields for ignore_errors option. */
+typedef struct SafeCopyFromState
+{
+#define REPLAY_BUFFER_SIZE 10
+ HeapTuple *replay_buffer; /* accumulates tuples for replaying it after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active; /* if true we replay tuples from buffer */
+ bool begin_subxact; /* if true we can begin subtransaction */
+ bool tuple_is_valid;
+ bool has_instead_insert_row_trig;
+
+ MemoryContext replay_cxt;
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -71,6 +90,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *sfcstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..d74575fd40 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,134 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int check (n != 6), m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: new row for relation "check_ign_err" violates check constraint "check_ign_err_n_check" Failing row contains (6, {6}, 6).
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: new row for relation "check_ign_err" violates check constraint "check_ign_err_n_check" Failing row contains (6, {6}, 6).
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: 5 errors
+SELECT * FROM trig_test;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 6 | {6} | 6
+ 8 | {8} | 8
+(3 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case is in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 5 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int check (n != 6), m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: new row for relation "check_ign_err_part2" violates check constraint "check_ign_err_n_check" Failing row contains (6, {6}, 6).
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..8d29ceba26 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int check (n != 6), m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+6 {6} 6
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+6 {6} 6
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+6 {6} 6
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM trig_test;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case is in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+6 {6} 6
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int check (n != 6), m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+ FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+6 {6} 6
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
[text/x-patch] 0006-COPY_IGNORE_ERRORS.patch (23.7K, ../../CALH1Lgsr7Asz6r6jnRta7DVEjKwyz=b3XJhyCMoYv8Yy9y4cXg@mail.gmail.com/4-0006-COPY_IGNORE_ERRORS.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index c25b52d0cb..22c992e6f6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows containing columns where the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 49924e476a..f41b25f26a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -406,6 +406,7 @@ ProcessCopyOptions(ParseState *pstate,
bool is_from,
List *options)
{
+ bool ignore_errors_specified = false;
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
@@ -448,6 +449,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index e8bb168aea..39f1dca084 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -535,6 +535,7 @@ CopyFrom(CopyFromState cstate)
ExprContext *econtext;
TupleTableSlot *singleslot = NULL;
MemoryContext oldcontext = CurrentMemoryContext;
+ ResourceOwner oldowner = CurrentResourceOwner;
PartitionTupleRouting *proute = NULL;
ErrorContextCallback errcallback;
@@ -819,6 +820,26 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option*/
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->sfcstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->sfcstate->replay_cxt = AllocSetContextCreate(oldcontext,
+ "Replay_context",
+ ALLOCSET_DEFAULT_SIZES);
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ cstate->sfcstate->saved_tuples = 0;
+ cstate->sfcstate->replayed_tuples = 0;
+ cstate->sfcstate->errors = 0;
+ cstate->sfcstate->replay_is_active = false;
+ cstate->sfcstate->begin_subxact = true;
+
+ cstate->sfcstate->oldowner = oldowner;
+ cstate->sfcstate->oldcontext = oldcontext;
+ }
+
for (;;)
{
TupleTableSlot *myslot;
@@ -855,9 +876,18 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ if (cstate->sfcstate)
+ {
+ /* If option IGNORE_ERRORS is enabled, COPY skips rows with errors */
+ if (!safeNextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
+ if (!cstate->sfcstate->replay_is_active)
+ continue;
+ }
+ else
+ /* Directly store the values/nulls array in the slot */
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
ExecStoreVirtualTuple(myslot);
@@ -1035,7 +1065,21 @@ CopyFrom(CopyFromState cstate)
*/
if (resultRelInfo->ri_FdwRoutine == NULL &&
resultRelInfo->ri_RelationDesc->rd_att->constr)
- ExecConstraints(resultRelInfo, myslot, estate);
+ {
+ if (cstate->opts.ignore_errors)
+ {
+ PG_TRY();
+ ExecConstraints(resultRelInfo, myslot, estate);
+ PG_CATCH();
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = cstate->sfcstate->oldowner;
+
+ PG_RE_THROW();
+ PG_END_TRY();
+ }
+ else
+ ExecConstraints(resultRelInfo, myslot, estate);
+ }
/*
* Also check the tuple against the partition constraint, if
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 7cf3e865cf..6b92e781e3 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -839,6 +839,169 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
return true;
}
+/*
+ * Analog of NextCopyFrom() but ignores rows with errors while copying.
+ */
+bool
+safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+
+ PG_TRY();
+ {
+ if (sfcstate->begin_subxact)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = false;
+ }
+
+ if (!sfcstate->replay_is_active)
+ {
+ if (sfcstate->saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ MemoryContext cxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+ bool valid_row = NextCopyFrom(cstate, econtext, values, nulls);
+
+ CurrentMemoryContext = cxt;
+
+ if (valid_row)
+ {
+ /* Filling replay_buffer in Replay_context */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+ HeapTuple saved_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), values, nulls);
+
+ sfcstate->replay_buffer[sfcstate->saved_tuples++] = saved_tuple;
+ MemoryContextSwitchTo(cxt);
+ }
+ else if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ /* Prepare for replaying remaining tuples if they exist */
+ sfcstate->replay_is_active = true;
+ else
+ {
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ if (sfcstate->errors == 0)
+ ereport(NOTICE,
+ errmsg("%d errors", sfcstate->errors));
+ else if (sfcstate->errors == 1)
+ ereport(WARNING,
+ errmsg("%d error", sfcstate->errors));
+ else
+ ereport(WARNING,
+ errmsg("%d errors", sfcstate->errors));
+
+ return false;
+ }
+ }
+ else
+ /* Buffer was filled, prepare for replaying */
+ sfcstate->replay_is_active = true;
+ }
+
+ if (sfcstate->replay_is_active)
+ {
+ if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ /* Replaying the tuple */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+
+ heap_deform_tuple(sfcstate->replay_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel), values, nulls);
+ MemoryContextSwitchTo(cxt);
+ }
+ else
+ {
+ /* Clean up replay_buffer */
+ MemoryContextReset(sfcstate->replay_cxt);
+ sfcstate->saved_tuples = sfcstate->replayed_tuples = 0;
+
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->begin_subxact = true;
+ sfcstate->replay_is_active = false;
+ }
+ }
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore data exceptions */
+ case ERRCODE_CHARACTER_NOT_IN_REPERTOIRE:
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+
+ sfcstate->begin_subxact = true;
+
+ break;
+ default:
+ PG_RE_THROW();
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ return true;
+}
+
/*
* Read next tuple from file for COPY FROM. Return false if no more tuples.
*
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b5ab9d9c9a..b49954c0aa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -808,7 +808,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3477,6 +3477,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeBoolean(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17778,6 +17782,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18357,6 +18362,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 62a39779b9..fe590ff7a8 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2748,7 +2748,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..006e1024e1 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
@@ -76,6 +77,8 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
const char *filename,
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
+extern bool safeNextCopyFrom(CopyFromState cstate, ExprContext *econtext,
+ Datum *values, bool *nulls);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Datum *values, bool *nulls);
extern bool NextCopyFromRawFields(CopyFromState cstate,
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index e37c6032ae..9100d5f247 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -49,6 +50,24 @@ typedef enum CopyInsertMethod
CIM_MULTI_CONDITIONAL /* use table_multi_insert only if valid */
} CopyInsertMethod;
+/*
+ * Struct that holding fields for ignore_errors option
+ */
+typedef struct SafeCopyFromState
+{
+#define REPLAY_BUFFER_SIZE 1000
+ HeapTuple *replay_buffer; /* accumulates tuples for replaying it after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active; /* if true we replay tuples from buffer */
+ bool begin_subxact; /* if true we can begin subtransaction */
+
+ MemoryContext replay_cxt;
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -71,6 +90,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *sfcstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..acf4917e64 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,135 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM trig_test;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case is in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..b25b20182e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM trig_test;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case is in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-09-26 13:04 ` torikoshia <[email protected]>
2022-09-29 13:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2022-09-26 13:04 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers; [email protected]
On 2022-09-21 21:11, Damir Belyalov wrote:
Thanks for updating patch.
> In the previous patch there was an error when processing constraints.
> The patch was fixed, but the code grew up and became more complicated
> (0005-COPY_IGNORE_ERRORS). I also simplified the logic of
> safeNextCopyFrom().
> You asked why we need subtransactions, so the answer is in this patch.
> When processing a row that does not satisfy constraints or INSTEAD OF
> triggers, it is necessary to rollback the subtransaction and return
> the table to its original state.
> Cause of complexity, I had to abandon the constraints, triggers
> processing in and handle only errors that occur when reading the file.
> Attaching simplified patch (0006-COPY_IGNORE_ERRORS).
Do you mean you stop dealing with errors concerned with constraints and
triggers and we should review 0006-COPY_IGNORE_ERRORS?
> Tried to implement your error and could not. The result was the same
> as COPY FROM implements.
Hmm, I applied v6 patch and when canceled COPY by sending SIGINT(ctrl +
C), I faced the same situation as below.
I tested it on CentOS 8.4.
=# COPY test FROM '/home/tori/pgsql/master/10000000.data' WITH
(IGNORE_ERRORS);
^CCancel request sent
ERROR: canceling statement due to user request
CONTEXT: COPY test, line 628000: "628000 xxx"
=# SELECT 1;
ERROR: current transaction is aborted, commands ignored until end of
transaction block
=# ABORT;
FATAL: UserAbortTransactionBlock: unexpected state STARTED
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.
I did the same procedure on COPY FROM without IGNORE_ERRORS and didn't
face this situation.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2022-09-29 13:18 ` Damir Belyalov <[email protected]>
2022-10-17 10:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-09-29 13:18 UTC (permalink / raw)
To: torikoshia <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
>
> Do you mean you stop dealing with errors concerned with constraints and
> triggers and we should review 0006-COPY_IGNORE_ERRORS?
>
Yes, this patch is simpler and I think it's worth adding it first.
> Hmm, I applied v6 patch and when canceled COPY by sending SIGINT(ctrl +
> C), I faced the same situation as below.
> I tested it on CentOS 8.4.
>
Thank you for pointing out this error. it really needs to be taken into
account. In the previous 0006 patch, there were 2 stages in one
subtransaction - filling the buffer and 'replay mode' (reading from the
buffer). Since only NextCopyFrom() is included in PG_TRY(), and the
ERRCODE_QUERY_CANCELED error can occur anywhere, it is impossible to catch
it and rollback the subtransaction.
I changed the 0006 patch and fixed this error and now only the 'replay
buffer filling' is made in the subtransaction.
Patch 0005 (that processed constraints) needs to be finalized, because it
requires subtransactions to rollback constraints, triggers. Therefore, it
is not possible to fix it yet. There is a decision to put for(;;) loop in
PG_TRY. It will solve the problem, but the code will be too complicated.
Attachments:
[text/x-patch] 0007-COPY_IGNORE_ERRORS.patch (22.5K, ../../CALH1LguQ5P0uZSYQFy4gJihvV11EkLx+gwxZ-+1DFq1Z6dJqbg@mail.gmail.com/3-0007-COPY_IGNORE_ERRORS.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index c25b52d0cb..22c992e6f6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows containing columns where the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 49924e476a..f41b25f26a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -406,6 +406,7 @@ ProcessCopyOptions(ParseState *pstate,
bool is_from,
List *options)
{
+ bool ignore_errors_specified = false;
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
@@ -448,6 +449,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index e8bb168aea..caa3375758 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -106,6 +106,9 @@ static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static bool FillReplayBuffer(CopyFromState cstate, ExprContext *econtext,
+ TupleTableSlot *myslot);
+
/*
* error context callback for COPY FROM
*
@@ -521,6 +524,177 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Fills replay_buffer for safe copying, enables by IGNORE_ERRORS option.
+ */
+bool
+FillReplayBuffer(CopyFromState cstate, ExprContext *econtext, TupleTableSlot *myslot)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+ bool valid_row = true;
+
+ if (!sfcstate->replay_is_active)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ while (sfcstate->saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ bool tuple_is_valid = false;
+
+ PG_TRY();
+ {
+ MemoryContext cxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+
+ valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+
+ if (valid_row)
+ tuple_is_valid = true;
+
+ CurrentMemoryContext = cxt;
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ Assert(IsSubTransaction());
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore data exceptions */
+ case ERRCODE_CHARACTER_NOT_IN_REPERTOIRE:
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ /* If the error can be processed, begin a new subtransaction */
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+ break;
+ default:
+ PG_RE_THROW();
+ break;
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ if (tuple_is_valid)
+ {
+ /* Filling replay_buffer in Replay_context */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+ HeapTuple saved_tuple;
+
+ saved_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+ sfcstate->replay_buffer[sfcstate->saved_tuples++] = saved_tuple;
+
+ MemoryContextSwitchTo(cxt);
+ }
+
+ MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+ ExecClearTuple(myslot);
+
+ if (!valid_row)
+ break;
+ }
+
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ /* End of file or buffer was filled, prepare to replay remaining tuples from buffer */
+ sfcstate->replay_is_active = true;
+ }
+
+ if (sfcstate->replay_is_active)
+ {
+ if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ /* Replaying the tuple */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+
+ heap_deform_tuple(sfcstate->replay_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel),
+ myslot->tts_values, myslot->tts_isnull);
+ MemoryContextSwitchTo(cxt);
+ }
+ else
+ {
+ /* All tuples from buffer were replayed, clean it up */
+ MemoryContextReset(sfcstate->replay_cxt);
+ sfcstate->saved_tuples = sfcstate->replayed_tuples = 0;
+
+ sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ sfcstate->replay_is_active = false;
+
+ if (!valid_row)
+ {
+ /* All tuples were replayed */
+ if (sfcstate->errors == 0)
+ ereport(NOTICE,
+ errmsg("%d errors", sfcstate->errors));
+ else if (sfcstate->errors == 1)
+ ereport(WARNING,
+ errmsg("%d error", sfcstate->errors));
+ else
+ ereport(WARNING,
+ errmsg("%d errors", sfcstate->errors));
+
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -855,9 +1029,19 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ if (cstate->sfcstate)
+ {
+ /* If option IGNORE_ERRORS is enabled, COPY skips rows with errors */
+ if (!FillReplayBuffer(cstate, econtext, myslot))
+ break;
+
+ if (!cstate->sfcstate->replay_is_active)
+ continue;
+ }
+ else
+ /* Directly store the values/nulls array in the slot */
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
ExecStoreVirtualTuple(myslot);
@@ -1550,6 +1734,25 @@ BeginCopyFrom(ParseState *pstate,
cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
}
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option */
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->sfcstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->sfcstate->replay_cxt = AllocSetContextCreate(oldcontext,
+ "Replay_context",
+ ALLOCSET_DEFAULT_SIZES);
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ cstate->sfcstate->saved_tuples = 0;
+ cstate->sfcstate->replayed_tuples = 0;
+ cstate->sfcstate->errors = 0;
+ cstate->sfcstate->replay_is_active = false;
+
+ cstate->sfcstate->oldowner = CurrentResourceOwner;
+ cstate->sfcstate->oldcontext = cstate->copycontext;
+ }
+
MemoryContextSwitchTo(oldcontext);
return cstate;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b5ab9d9c9a..b49954c0aa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -808,7 +808,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3477,6 +3477,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeBoolean(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -17778,6 +17782,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -18357,6 +18362,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 62a39779b9..fe590ff7a8 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2748,7 +2748,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index cb0096aeb6..2b696f99bc 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index e37c6032ae..d3f4c8d9df 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -49,6 +50,23 @@ typedef enum CopyInsertMethod
CIM_MULTI_CONDITIONAL /* use table_multi_insert only if valid */
} CopyInsertMethod;
+/*
+ * Struct that holding fields for ignore_errors option
+ */
+typedef struct SafeCopyFromState
+{
+#define REPLAY_BUFFER_SIZE 1000
+ HeapTuple *replay_buffer; /* accumulates tuples for replaying it after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active; /* if true we replay tuples from buffer */
+
+ MemoryContext replay_cxt;
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -71,6 +89,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *sfcstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index ae35f03251..2af11bd359 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -201,6 +201,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..acf4917e64 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,135 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM trig_test;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case is in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..b25b20182e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM trig_test;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case is in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-29 13:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-10-17 10:17 ` Damir Belyalov <[email protected]>
2022-11-02 08:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2022-10-17 10:17 UTC (permalink / raw)
To: torikoshia <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Updated the patch due to conflicts when applying to master.
>
Attachments:
[text/x-patch] 0008-COPY_IGNORE_ERRORS.patch (22.5K, ../../CALH1Lgug+mR9A1bBzR8hrp2oW4=jvR_N0An-i2X_j_y9C1fkiA@mail.gmail.com/3-0008-COPY_IGNORE_ERRORS.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index c25b52d0cb..22c992e6f6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows containing columns where the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index db4c9dbc23..d04753a4c8 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -406,6 +406,7 @@ ProcessCopyOptions(ParseState *pstate,
bool is_from,
List *options)
{
+ bool ignore_errors_specified = false;
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
@@ -448,6 +449,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index a079c70152..fa169d2cf4 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -107,6 +107,9 @@ static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static bool FillReplayBuffer(CopyFromState cstate, ExprContext *econtext,
+ TupleTableSlot *myslot);
+
/*
* error context callback for COPY FROM
*
@@ -625,6 +628,177 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Fills replay_buffer for safe copying, enables by IGNORE_ERRORS option.
+ */
+bool
+FillReplayBuffer(CopyFromState cstate, ExprContext *econtext, TupleTableSlot *myslot)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+ bool valid_row = true;
+
+ if (!sfcstate->replay_is_active)
+ {
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ while (sfcstate->saved_tuples < REPLAY_BUFFER_SIZE)
+ {
+ bool tuple_is_valid = false;
+
+ PG_TRY();
+ {
+ MemoryContext cxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+
+ valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+
+ if (valid_row)
+ tuple_is_valid = true;
+
+ CurrentMemoryContext = cxt;
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ Assert(IsSubTransaction());
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore data exceptions */
+ case ERRCODE_CHARACTER_NOT_IN_REPERTOIRE:
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ /* If the error can be processed, begin a new subtransaction */
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+ break;
+ default:
+ PG_RE_THROW();
+ break;
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ if (tuple_is_valid)
+ {
+ /* Filling replay_buffer in Replay_context */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+ HeapTuple saved_tuple;
+
+ saved_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+ sfcstate->replay_buffer[sfcstate->saved_tuples++] = saved_tuple;
+
+ MemoryContextSwitchTo(cxt);
+ }
+
+ MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+ ExecClearTuple(myslot);
+
+ if (!valid_row)
+ break;
+ }
+
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ /* End of file or buffer was filled, prepare to replay remaining tuples from buffer */
+ sfcstate->replay_is_active = true;
+ }
+
+ if (sfcstate->replay_is_active)
+ {
+ if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ /* Replaying the tuple */
+ MemoryContext cxt = MemoryContextSwitchTo(sfcstate->replay_cxt);
+
+ heap_deform_tuple(sfcstate->replay_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel),
+ myslot->tts_values, myslot->tts_isnull);
+ MemoryContextSwitchTo(cxt);
+ }
+ else
+ {
+ /* All tuples from buffer were replayed, clean it up */
+ MemoryContextReset(sfcstate->replay_cxt);
+ sfcstate->saved_tuples = sfcstate->replayed_tuples = 0;
+
+ sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ sfcstate->replay_is_active = false;
+
+ if (!valid_row)
+ {
+ /* All tuples were replayed */
+ if (sfcstate->errors == 0)
+ ereport(NOTICE,
+ errmsg("%d errors", sfcstate->errors));
+ else if (sfcstate->errors == 1)
+ ereport(WARNING,
+ errmsg("%d error", sfcstate->errors));
+ else
+ ereport(WARNING,
+ errmsg("%d errors", sfcstate->errors));
+
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -985,9 +1159,19 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
- break;
+ if (cstate->sfcstate)
+ {
+ /* If option IGNORE_ERRORS is enabled, COPY skips rows with errors */
+ if (!FillReplayBuffer(cstate, econtext, myslot))
+ break;
+
+ if (!cstate->sfcstate->replay_is_active)
+ continue;
+ }
+ else
+ /* Directly store the values/nulls array in the slot */
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ break;
ExecStoreVirtualTuple(myslot);
@@ -1695,6 +1879,25 @@ BeginCopyFrom(ParseState *pstate,
cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
}
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option */
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->sfcstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->sfcstate->replay_cxt = AllocSetContextCreate(oldcontext,
+ "Replay_context",
+ ALLOCSET_DEFAULT_SIZES);
+ cstate->sfcstate->replay_buffer = MemoryContextAlloc(cstate->sfcstate->replay_cxt,
+ REPLAY_BUFFER_SIZE * sizeof(HeapTuple));
+ cstate->sfcstate->saved_tuples = 0;
+ cstate->sfcstate->replayed_tuples = 0;
+ cstate->sfcstate->errors = 0;
+ cstate->sfcstate->replay_is_active = false;
+
+ cstate->sfcstate->oldowner = CurrentResourceOwner;
+ cstate->sfcstate->oldcontext = cstate->copycontext;
+ }
+
MemoryContextSwitchTo(oldcontext);
return cstate;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 737bd2d06d..b3a6c9931e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -702,7 +702,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3359,6 +3359,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeBoolean(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -16756,6 +16760,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17310,6 +17315,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 584d9d5ae6..33d583a94c 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2757,7 +2757,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index b77b935005..0bf9641b6e 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 8d9cc5accd..3289d96872 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -52,6 +53,23 @@ typedef enum CopyInsertMethod
* ExecForeignBatchInsert only if valid */
} CopyInsertMethod;
+/*
+ * Struct that holding fields for ignore_errors option
+ */
+typedef struct SafeCopyFromState
+{
+#define REPLAY_BUFFER_SIZE 1000
+ HeapTuple *replay_buffer; /* accumulates tuples for replaying it after an error */
+ int saved_tuples; /* # of tuples in replay_buffer */
+ int replayed_tuples; /* # of tuples was replayed from buffer */
+ int errors; /* total # of errors */
+ bool replay_is_active; /* if true we replay tuples from buffer */
+
+ MemoryContext replay_cxt;
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -74,6 +92,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *sfcstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 957ee18d84..ed25a8c86c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -196,6 +196,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..acf4917e64 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,135 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM trig_test;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case is in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: 6 errors
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..b25b20182e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM trig_test;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case is in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-29 13:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-10-17 10:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-11-02 08:46 ` Damir Belyalov <[email protected]>
2022-12-07 11:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Nikita Malakhov <[email protected]>
2022-12-09 14:25 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Danil Anisimow <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: Damir Belyalov @ 2022-11-02 08:46 UTC (permalink / raw)
To: torikoshia <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Updated the patch:
- Optimized and simplified logic of IGNORE_ERRORS
- Changed variable names to more understandable ones
- Added an analogue of MAX_BUFFERED_BYTES for safe buffer
Regards,
Damir Belyalov
Postgres Professional
>
Attachments:
[text/x-patch] 0010-COPY_IGNORE_ERRORS.patch (22.3K, ../../CALH1LguPAN=p5wHKwGcHrjE8Vj1x4o1z3Pk228YDif6LFzbeYQ@mail.gmail.com/3-0010-COPY_IGNORE_ERRORS.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index c25b52d0cb..22c992e6f6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -34,6 +34,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORMAT <replaceable class="parameter">format_name</replaceable>
FREEZE [ <replaceable class="parameter">boolean</replaceable> ]
+ IGNORE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
DELIMITER '<replaceable class="parameter">delimiter_character</replaceable>'
NULL '<replaceable class="parameter">null_string</replaceable>'
HEADER [ <replaceable class="parameter">boolean</replaceable> | MATCH ]
@@ -233,6 +234,19 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ containing syntax errors in data, rows with too many or too few columns,
+ rows containing columns where the data type's input function raises an error.
+ Outputs warnings about rows with incorrect data (the number of warnings
+ is not more than 100) and the total number of errors.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>DELIMITER</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index db4c9dbc23..d04753a4c8 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -406,6 +406,7 @@ ProcessCopyOptions(ParseState *pstate,
bool is_from,
List *options)
{
+ bool ignore_errors_specified = false;
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
@@ -448,6 +449,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_errors") == 0)
+ {
+ if (ignore_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_errors_specified = true;
+ opts_out->ignore_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index a079c70152..846eac022d 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -107,6 +107,9 @@ static char *limit_printout_length(const char *str);
static void ClosePipeFromProgram(CopyFromState cstate);
+static bool SafeCopying(CopyFromState cstate, ExprContext *econtext,
+ TupleTableSlot *myslot);
+
/*
* error context callback for COPY FROM
*
@@ -625,6 +628,175 @@ CopyMultiInsertInfoStore(CopyMultiInsertInfo *miinfo, ResultRelInfo *rri,
miinfo->bufferedBytes += tuplen;
}
+/*
+ * Safely reads source data, converts to a tuple and fills tuple buffer.
+ * Skips some data in the case of failed conversion if data source for
+ * a next tuple can be surely read without a danger.
+ */
+bool
+SafeCopying(CopyFromState cstate, ExprContext *econtext, TupleTableSlot *myslot)
+{
+ SafeCopyFromState *sfcstate = cstate->sfcstate;
+ bool valid_row = true;
+
+ /* Standard COPY if IGNORE_ERRORS is disabled */
+ if (!cstate->sfcstate)
+ /* Directly stores the values/nulls array in the slot */
+ return NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+
+ if (sfcstate->replayed_tuples < sfcstate->saved_tuples)
+ {
+ Assert(sfcstate->saved_tuples > 0);
+
+ /* Prepare to replay the tuple */
+ heap_deform_tuple(sfcstate->safe_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel),
+ myslot->tts_values, myslot->tts_isnull);
+ return true;
+ }
+ else
+ {
+ /* All tuples from buffer were replayed, clean it up */
+ MemoryContextReset(sfcstate->safe_cxt);
+
+ sfcstate->saved_tuples = sfcstate->replayed_tuples = 0;
+ sfcstate->safeBufferBytes = 0;
+ }
+
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ while (sfcstate->saved_tuples < SAFE_BUFFER_SIZE &&
+ sfcstate->safeBufferBytes < MAX_SAFE_BUFFER_BYTES)
+ {
+ bool tuple_is_valid = true;
+
+ PG_TRY();
+ {
+ MemoryContext cxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
+
+ valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull);
+ tuple_is_valid = valid_row;
+
+ if (valid_row)
+ sfcstate->safeBufferBytes += cstate->line_buf.len;
+
+ CurrentMemoryContext = cxt;
+ }
+ PG_CATCH();
+ {
+ MemoryContext ecxt = MemoryContextSwitchTo(sfcstate->oldcontext);
+ ErrorData *errdata = CopyErrorData();
+
+ tuple_is_valid = false;
+
+ Assert(IsSubTransaction());
+
+ RollbackAndReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ switch (errdata->sqlerrcode)
+ {
+ /* Ignore data exceptions */
+ case ERRCODE_CHARACTER_NOT_IN_REPERTOIRE:
+ case ERRCODE_DATA_EXCEPTION:
+ case ERRCODE_ARRAY_ELEMENT_ERROR:
+ case ERRCODE_DATETIME_VALUE_OUT_OF_RANGE:
+ case ERRCODE_INTERVAL_FIELD_OVERFLOW:
+ case ERRCODE_INVALID_CHARACTER_VALUE_FOR_CAST:
+ case ERRCODE_INVALID_DATETIME_FORMAT:
+ case ERRCODE_INVALID_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_ESCAPE_SEQUENCE:
+ case ERRCODE_NONSTANDARD_USE_OF_ESCAPE_CHARACTER:
+ case ERRCODE_INVALID_PARAMETER_VALUE:
+ case ERRCODE_INVALID_TABLESAMPLE_ARGUMENT:
+ case ERRCODE_INVALID_TIME_ZONE_DISPLACEMENT_VALUE:
+ case ERRCODE_NULL_VALUE_NOT_ALLOWED:
+ case ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE:
+ case ERRCODE_SEQUENCE_GENERATOR_LIMIT_EXCEEDED:
+ case ERRCODE_STRING_DATA_LENGTH_MISMATCH:
+ case ERRCODE_STRING_DATA_RIGHT_TRUNCATION:
+ case ERRCODE_INVALID_TEXT_REPRESENTATION:
+ case ERRCODE_INVALID_BINARY_REPRESENTATION:
+ case ERRCODE_BAD_COPY_FILE_FORMAT:
+ case ERRCODE_UNTRANSLATABLE_CHARACTER:
+ case ERRCODE_DUPLICATE_JSON_OBJECT_KEY_VALUE:
+ case ERRCODE_INVALID_ARGUMENT_FOR_SQL_JSON_DATETIME_FUNCTION:
+ case ERRCODE_INVALID_JSON_TEXT:
+ case ERRCODE_INVALID_SQL_JSON_SUBSCRIPT:
+ case ERRCODE_MORE_THAN_ONE_SQL_JSON_ITEM:
+ case ERRCODE_NO_SQL_JSON_ITEM:
+ case ERRCODE_NON_NUMERIC_SQL_JSON_ITEM:
+ case ERRCODE_NON_UNIQUE_KEYS_IN_A_JSON_OBJECT:
+ case ERRCODE_SINGLETON_SQL_JSON_ITEM_REQUIRED:
+ case ERRCODE_SQL_JSON_ARRAY_NOT_FOUND:
+ case ERRCODE_SQL_JSON_MEMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_NUMBER_NOT_FOUND:
+ case ERRCODE_SQL_JSON_OBJECT_NOT_FOUND:
+ case ERRCODE_TOO_MANY_JSON_ARRAY_ELEMENTS:
+ case ERRCODE_TOO_MANY_JSON_OBJECT_MEMBERS:
+ case ERRCODE_SQL_JSON_SCALAR_REQUIRED:
+ case ERRCODE_SQL_JSON_ITEM_CANNOT_BE_CAST_TO_TARGET_TYPE:
+ /* If the error can be processed, begin a new subtransaction */
+ BeginInternalSubTransaction(NULL);
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ sfcstate->errors++;
+ if (sfcstate->errors <= 100)
+ ereport(WARNING,
+ (errcode(errdata->sqlerrcode),
+ errmsg("%s", errdata->context)));
+ break;
+ default:
+ MemoryContextSwitchTo(ecxt);
+
+ PG_RE_THROW();
+
+ break;
+ }
+
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ MemoryContextSwitchTo(ecxt);
+ }
+ PG_END_TRY();
+
+ if (tuple_is_valid)
+ {
+ /* Add tuple to safe_buffer in Safe_context */
+ HeapTuple saved_tuple;
+
+ MemoryContextSwitchTo(sfcstate->safe_cxt);
+
+ saved_tuple = heap_form_tuple(RelationGetDescr(cstate->rel), myslot->tts_values, myslot->tts_isnull);
+ sfcstate->safe_buffer[sfcstate->saved_tuples++] = saved_tuple;
+ }
+
+ ExecClearTuple(myslot);
+
+ if (!valid_row)
+ break;
+ }
+
+ ReleaseCurrentSubTransaction();
+ CurrentResourceOwner = sfcstate->oldowner;
+
+ /* Prepare to replay the first tuple from safe_buffer */
+ if (sfcstate->saved_tuples != 0)
+ {
+ heap_deform_tuple(sfcstate->safe_buffer[sfcstate->replayed_tuples++], RelationGetDescr(cstate->rel),
+ myslot->tts_values, myslot->tts_isnull);
+ return true;
+ }
+
+ /* End of file and nothing to replay? */
+ if (!valid_row && sfcstate->replayed_tuples == sfcstate->saved_tuples)
+ return false;
+
+ return true;
+}
+
/*
* Copy FROM file to relation.
*/
@@ -985,8 +1157,8 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
- /* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ /* Standard copying with option "safe copying" enabled by IGNORE_ERRORS. */
+ if (!SafeCopying(cstate, econtext, myslot))
break;
ExecStoreVirtualTuple(myslot);
@@ -1270,6 +1442,10 @@ CopyFrom(CopyFromState cstate)
}
}
+ if (cstate->sfcstate && cstate->sfcstate->errors > 0)
+ ereport(WARNING,
+ errmsg("Errors: %d", cstate->sfcstate->errors));
+
/* Flush any remaining buffered tuples */
if (insertMethod != CIM_SINGLE)
{
@@ -1695,6 +1871,23 @@ BeginCopyFrom(ParseState *pstate,
cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
}
+ /* Initialize safeCopyFromState for IGNORE_ERRORS option */
+ if (cstate->opts.ignore_errors)
+ {
+ cstate->sfcstate = palloc(sizeof(SafeCopyFromState));
+
+ cstate->sfcstate->safe_cxt = AllocSetContextCreate(oldcontext,
+ "Safe_context",
+ ALLOCSET_DEFAULT_SIZES);
+ cstate->sfcstate->saved_tuples = 0;
+ cstate->sfcstate->replayed_tuples = 0;
+ cstate->sfcstate->safeBufferBytes = 0;
+ cstate->sfcstate->errors = 0;
+
+ cstate->sfcstate->oldowner = CurrentResourceOwner;
+ cstate->sfcstate->oldcontext = cstate->copycontext;
+ }
+
MemoryContextSwitchTo(oldcontext);
return cstate;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 737bd2d06d..b3a6c9931e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -702,7 +702,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
- IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
+ IDENTITY_P IF_P IGNORE_ERRORS ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -3359,6 +3359,10 @@ copy_opt_item:
{
$$ = makeDefElem("freeze", (Node *) makeBoolean(true), @1);
}
+ | IGNORE_ERRORS
+ {
+ $$ = makeDefElem("ignore_errors", (Node *)makeBoolean(true), @1);
+ }
| DELIMITER opt_as Sconst
{
$$ = makeDefElem("delimiter", (Node *) makeString($3), @1);
@@ -16756,6 +16760,7 @@ unreserved_keyword:
| HOUR_P
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| IMMEDIATE
| IMMUTABLE
| IMPLICIT_P
@@ -17310,6 +17315,7 @@ bare_label_keyword:
| HOLD
| IDENTITY_P
| IF_P
+ | IGNORE_ERRORS
| ILIKE
| IMMEDIATE
| IMMUTABLE
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 584d9d5ae6..33d583a94c 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2757,7 +2757,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING",
+ "IGNORE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index b77b935005..0bf9641b6e 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_errors; /* ignore rows with errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 8d9cc5accd..7c65157866 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "utils/resowner.h"
/*
* Represents the different source cases we need to worry about at
@@ -52,6 +53,25 @@ typedef enum CopyInsertMethod
* ExecForeignBatchInsert only if valid */
} CopyInsertMethod;
+/*
+ * Struct that holding fields for safe copying option enabled by IGNORE_ERRORS.
+ */
+typedef struct SafeCopyFromState
+{
+#define SAFE_BUFFER_SIZE 1000
+#define MAX_SAFE_BUFFER_BYTES 65535
+
+ HeapTuple safe_buffer[SAFE_BUFFER_SIZE]; /* accumulates valid tuples */
+ int saved_tuples; /* # of tuples in safe_buffer */
+ int replayed_tuples; /* # of tuples were replayed from buffer */
+ int safeBufferBytes; /* # of bytes from all buffered tuples */
+ int errors; /* total # of errors */
+
+ MemoryContext safe_cxt;
+ MemoryContext oldcontext;
+ ResourceOwner oldowner;
+} SafeCopyFromState;
+
/*
* This struct contains all the state variables used throughout a COPY FROM
* operation.
@@ -74,6 +94,7 @@ typedef struct CopyFromStateData
char *filename; /* filename, or NULL for STDIN */
bool is_program; /* is 'filename' a program to popen? */
copy_data_source_cb data_source_cb; /* function for reading data */
+ SafeCopyFromState *sfcstate; /* struct for ignore_errors option */
CopyFormatOptions opts;
bool *convert_select_flags; /* per-column CSV/TEXT CS flags */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 957ee18d84..ed25a8c86c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -196,6 +196,7 @@ PG_KEYWORD("hold", HOLD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("hour", HOUR_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("identity", IDENTITY_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("if", IF_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("ignore_errors", IGNORE_ERRORS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("immediate", IMMEDIATE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("immutable", IMMUTABLE, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 5f3685e9ef..cc6d572cf1 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -649,6 +649,135 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: Errors: 6
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: Errors: 6
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before on check_ign_err;
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err_view, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err_view, line 3: "3 {3}"
+WARNING: COPY check_ign_err_view, line 4, column n: "a"
+WARNING: COPY check_ign_err_view, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err_view, line 6, column n: ""
+WARNING: COPY check_ign_err_view, line 7, column m: "{a, 7}"
+WARNING: Errors: 6
+SELECT * FROM trig_test;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+-- foreign table case is in postgres_fdw extension
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: Errors: 6
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+(1 row)
+
+DROP TABLE check_ign_err;
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+WARNING: COPY check_ign_err, line 2: "2 {2} 2 2"
+WARNING: COPY check_ign_err, line 3: "3 {3}"
+WARNING: COPY check_ign_err, line 4, column n: "a"
+WARNING: COPY check_ign_err, line 5, column k: "5555555555"
+WARNING: COPY check_ign_err, line 6, column n: ""
+WARNING: COPY check_ign_err, line 7, column m: "{a, 7}"
+WARNING: Errors: 6
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 8 | {8} | 8
+(2 rows)
+
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index b3c16af48e..b25b20182e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -454,6 +454,122 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_ERRORS option
+-- CIM_MULTI case
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+
+-- CIM_SINGLE cases
+-- BEFORE row trigger
+TRUNCATE check_ign_err;
+CREATE TABLE trig_test(n int, m int[], k int);
+CREATE FUNCTION fn_trig_before () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before on check_ign_err;
+
+-- INSTEAD OF row trigger
+TRUNCATE check_ign_err;
+TRUNCATE trig_test;
+CREATE VIEW check_ign_err_view AS SELECT * FROM check_ign_err;
+CREATE FUNCTION fn_trig_instead_of () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m, NEW.k);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_instead_of INSTEAD OF INSERT ON check_ign_err_view
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_instead_of();
+COPY check_ign_err_view FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM trig_test;
+DROP TRIGGER trig_instead_of ON check_ign_err_view;
+DROP VIEW check_ign_err_view;
+
+-- foreign table case is in postgres_fdw extension
+
+-- volatile function in WHERE clause
+TRUNCATE check_ign_err;
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS
+ WHERE n = floor(random()*(1-1+1))+1; /* finds values equal 1 */
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TABLE check_ign_err;
+
+-- CIM_MULTI_CONDITIONAL case
+-- INSERT triggers for partition tables
+TRUNCATE trig_test;
+CREATE TABLE check_ign_err (n int, m int[], k int)
+ PARTITION BY RANGE (k);
+CREATE TABLE check_ign_err_part1 PARTITION OF check_ign_err
+ FOR VALUES FROM (1) TO (4);
+CREATE TABLE check_ign_err_part2 PARTITION OF check_ign_err
+ FOR VALUES FROM (4) TO (9);
+CREATE FUNCTION fn_trig_before_part () RETURNS TRIGGER AS '
+ BEGIN
+ INSERT INTO trig_test VALUES(NEW.n, NEW.m);
+ RETURN NEW;
+ END;
+' LANGUAGE plpgsql;
+CREATE TRIGGER trig_before_part BEFORE INSERT ON check_ign_err
+FOR EACH ROW EXECUTE PROCEDURE fn_trig_before_part();
+COPY check_ign_err FROM STDIN WITH IGNORE_ERRORS WHERE n < 9;
+1 {1} 1
+2 {2} 2 2
+3 {3}
+a {4} 4
+5 {5} 5555555555
+
+7 {a, 7} 7
+8 {8} 8
+\.
+SELECT * FROM check_ign_err;
+DROP TRIGGER trig_before_part on check_ign_err;
+DROP TABLE trig_test;
+DROP TABLE check_ign_err CASCADE;
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-29 13:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-10-17 10:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-11-02 08:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-12-07 11:15 ` Nikita Malakhov <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Nikita Malakhov @ 2022-12-07 11:15 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: torikoshia <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi Damir!
Your work looks like a very promising feature for production systems,
where data often needs to be loaded from external sources.
I've looked over the discussion and want to make a proposal -
when we load a bunch of records in database it does not make sense
to output errors to command output, and does not make sense to limit
error output to any number at all, because if we decided to load data
anyway - we would want to have a list (a file) with all records that were
discarded because of errors, with related error information, to, say,
deal with errors and process these records later. It looks like a reasonable
addition to your patch.
As a command output some limited number of error messages has much
less meaning than overall stats - records processed, records loaded,
records discarded, total number of errors.
For example you can look the Oracle SQL Loader feature, I hope this could
give some ideas for further improvements.
On Wed, Nov 2, 2022 at 11:46 AM Damir Belyalov <[email protected]> wrote:
> Updated the patch:
> - Optimized and simplified logic of IGNORE_ERRORS
> - Changed variable names to more understandable ones
> - Added an analogue of MAX_BUFFERED_BYTES for safe buffer
>
>
> Regards,
> Damir Belyalov
> Postgres Professional
>
>>
--
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-15 12:29 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-15 13:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-22 12:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-08-24 16:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-08-29 13:02 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-21 12:11 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-09-26 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2022-09-29 13:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-10-17 10:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2022-11-02 08:46 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2022-12-09 14:25 ` Danil Anisimow <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Danil Anisimow @ 2022-12-09 14:25 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: torikoshia <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
Hi!
I have looked at your patch and have a few questions.
110: static bool SafeCopying(CopyFromState cstate, ExprContext *econtext,
111: TupleTableSlot *myslot);
---
636: bool
637: SafeCopying(CopyFromState cstate, ExprContext *econtext,
TupleTableSlot *myslot)
Why is there no static keyword in the definition of the SafeCopying()
function, but it is in the function declaration.
675: MemoryContext cxt =
MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);
676:
677: valid_row = NextCopyFrom(cstate, econtext, myslot->tts_values,
myslot->tts_isnull);
678: tuple_is_valid = valid_row;
679:
680: if (valid_row)
681: sfcstate->safeBufferBytes += cstate->line_buf.len;
682:
683: CurrentMemoryContext = cxt;
Why are you using a direct assignment to CurrentMemoryContext instead of
using the MemoryContextSwitchTo function in the SafeCopying() routine?
1160: /* Standard copying with option "safe copying" enabled by
IGNORE_ERRORS. */
1161: if (!SafeCopying(cstate, econtext, myslot))
1162: break;
I checked with GDB that the CurrentMemoryContext changes when SafeCopying
returns. And the target context may be different each time you do a COPY in
psql.
1879: cstate->sfcstate->safe_cxt = AllocSetContextCreate(oldcontext,
1880: "Safe_context",
1881: ALLOCSET_DEFAULT_SIZES);
When you initialize the cstate->sfcstate structure, you create a
cstate->sfcstate->safe_cxt memory context that inherits from oldcontext.
Was it intended to use cstate->copycontext as the parent context here?
On Wed, Nov 2, 2022 at 11:46 AM Damir Belyalov <[email protected]> wrote:
> Updated the patch:
> - Optimized and simplified logic of IGNORE_ERRORS
> - Changed variable names to more understandable ones
> - Added an analogue of MAX_BUFFERED_BYTES for safe buffer
>
>
> Regards,
> Damir Belyalov
> Postgres Professional
>
Regards,
Daniil Anisimov
Postgres Professional
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v15 10/10] pg_ls_* to show file type and show special files
@ 2020-03-31 19:40 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Justin Pryzby @ 2020-03-31 19:40 UTC (permalink / raw)
---
src/backend/utils/adt/genfile.c | 52 ++++++++++++--------
src/include/catalog/pg_proc.dat | 38 +++++++-------
src/test/regress/expected/misc_functions.out | 40 +++++++--------
src/test/regress/output/tablespace.source | 8 +--
src/test/regress/sql/misc_functions.sql | 6 +--
5 files changed, 77 insertions(+), 67 deletions(-)
diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c
index 4d90f37a21..1fdb2bf280 100644
--- a/src/backend/utils/adt/genfile.c
+++ b/src/backend/utils/adt/genfile.c
@@ -38,7 +38,7 @@
static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags);
-#define LS_DIR_ISDIR (1<<0) /* Show column: isdir */
+#define LS_DIR_TYPE (1<<0) /* Show column: type */
#define LS_DIR_METADATA (1<<1) /* Show columns: mtime, size */
#define LS_DIR_MISSING_OK (1<<2) /* Ignore ENOENT if the toplevel dir is missing */
#define LS_DIR_SKIP_DOT_DIRS (1<<3) /* Do not show . or .. */
@@ -53,7 +53,7 @@ static Datum pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags
#define LS_DIR_HISTORIC (LS_DIR_SKIP_DIRS|LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
/* Shortcut for common behavior */
-#define LS_DIR_COMMON (LS_DIR_SKIP_HIDDEN|LS_DIR_SKIP_SPECIAL|LS_DIR_METADATA)
+#define LS_DIR_COMMON (LS_DIR_SKIP_HIDDEN|LS_DIR_METADATA)
/*
* Convert a "text" filename argument to C string, and check it's allowable.
@@ -367,6 +367,26 @@ pg_read_binary_file_all(PG_FUNCTION_ARGS)
return pg_read_binary_file(fcinfo);
}
+/* Return a character indicating the type of file, or '?' if unknown type */
+static char
+get_file_type(mode_t mode, const char *path)
+{
+ if (S_ISREG(mode))
+ return '-';
+
+ if (S_ISDIR(mode))
+ return 'd';
+#ifndef WIN32
+ if (S_ISLNK(mode))
+ return 'l';
+#else
+ if (pgwin32_is_junction(path))
+ return 'l';
+#endif
+
+ return '?';
+}
+
/*
* stat a file
*/
@@ -414,7 +434,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
TupleDescInitEntry(tupdesc, (AttrNumber) 5,
"creation", TIMESTAMPTZOID, -1, 0);
TupleDescInitEntry(tupdesc, (AttrNumber) 6,
- "isdir", BOOLOID, -1, 0);
+ "type", CHAROID, -1, 0);
BlessTupleDesc(tupdesc);
memset(isnull, false, sizeof(isnull));
@@ -430,12 +450,7 @@ pg_stat_file(PG_FUNCTION_ARGS)
isnull[3] = true;
values[4] = TimestampTzGetDatum(time_t_to_timestamptz(fst.st_ctime));
#endif
- values[5] = BoolGetDatum(S_ISDIR(fst.st_mode));
-#ifdef WIN32
- /* Links should have isdir=false */
- if (pgwin32_is_junction(filename))
- values[5] = BoolGetDatum(false);
-#endif
+ values[5] = CharGetDatum(get_file_type(fst.st_mode, filename));
tuple = heap_form_tuple(tupdesc, values, isnull);
@@ -501,10 +516,10 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
MemoryContext oldcontext;
TypeFuncClass tuptype ;
- /* isdir depends on metadata */
- Assert(!(flags&LS_DIR_ISDIR) || (flags&LS_DIR_METADATA));
- /* Unreasonable to show isdir and skip dirs */
- Assert(!(flags&LS_DIR_ISDIR) || !(flags&LS_DIR_SKIP_DIRS));
+ /* type depends on metadata */
+ Assert(!(flags&LS_DIR_TYPE) || (flags&LS_DIR_METADATA));
+ /* Unreasonable to show type and skip dirs XXX */
+ Assert(!(flags&LS_DIR_TYPE) || !(flags&LS_DIR_SKIP_DIRS));
/* check the optional arguments */
if (PG_NARGS() == 3)
@@ -631,12 +646,7 @@ pg_ls_dir_files(FunctionCallInfo fcinfo, const char *dir, int flags)
nulls[4] = true;
values[5] = TimestampTzGetDatum(time_t_to_timestamptz(attrib.st_ctime));
#endif
- values[6] = BoolGetDatum(S_ISDIR(attrib.st_mode));
-#ifdef WIN32
- /* Links should have isdir=false */
- if (pgwin32_is_junction(path))
- values[6] = BoolGetDatum(false);
-#endif
+ values[6] = CharGetDatum(get_file_type(attrib.st_mode, path));
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -718,7 +728,7 @@ pg_ls_dir_metadata(PG_FUNCTION_ARGS)
char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
return pg_ls_dir_files(fcinfo, dirname,
- LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+ LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
}
/*
@@ -733,5 +743,5 @@ pg_ls_dir_metadata_1arg(PG_FUNCTION_ARGS)
char *dirname = convert_and_check_filename(PG_GETARG_TEXT_PP(0));
return pg_ls_dir_files(fcinfo, dirname,
- LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_ISDIR);
+ LS_DIR_METADATA | LS_DIR_SKIP_SPECIAL | LS_DIR_TYPE);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d97ff8097f..8f19220456 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6107,16 +6107,16 @@
{ oid => '2623', descr => 'get information about file',
proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
proargtypes => 'text',
- proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+ proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
proargmodes => '{i,o,o,o,o,o,o}',
- proargnames => '{filename,size,access,modification,change,creation,isdir}',
+ proargnames => '{filename,size,access,modification,change,creation,type}',
prosrc => 'pg_stat_file_1arg' },
{ oid => '3307', descr => 'get information about file',
proname => 'pg_stat_file', provolatile => 'v', prorettype => 'record',
proargtypes => 'text bool',
- proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
+ proallargtypes => '{text,bool,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
proargmodes => '{i,i,o,o,o,o,o,o}',
- proargnames => '{filename,missing_ok,size,access,modification,change,creation,isdir}',
+ proargnames => '{filename,missing_ok,size,access,modification,change,creation,type}',
prosrc => 'pg_stat_file' },
{ oid => '2624', descr => 'read text from a file',
proname => 'pg_read_file', provolatile => 'v', prorettype => 'text',
@@ -10777,13 +10777,13 @@
{ oid => '3353', descr => 'list files in the log directory',
proname => 'pg_ls_logdir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
- proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_logdir' },
+ proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_logdir' },
{ oid => '3354', descr => 'list of files in the WAL directory',
proname => 'pg_ls_waldir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
- proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_waldir' },
+ proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_waldir' },
{ oid => '5031', descr => 'list of files in the archive_status directory',
proname => 'pg_ls_archive_statusdir', procost => '10', prorows => '20',
proretset => 't', provolatile => 'v', prorettype => 'record',
@@ -10793,32 +10793,32 @@
{ oid => '5029', descr => 'list files in the pgsql_tmp directory',
proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{o,o,o,o,o,o,o}',
- proargnames => '{name,size,access,modification,change,creation,isdir}', prosrc => 'pg_ls_tmpdir_noargs' },
+ proallargtypes => '{text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{name,size,access,modification,change,creation,type}', prosrc => 'pg_ls_tmpdir_noargs' },
{ oid => '5030', descr => 'list files in the pgsql_tmp directory',
proname => 'pg_ls_tmpdir', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => 'oid',
- proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
- proargnames => '{tablespace,name,size,access,modification,change,creation,isdir}',
+ proallargtypes => '{oid,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+ proargnames => '{tablespace,name,size,access,modification,change,creation,type}',
prosrc => 'pg_ls_tmpdir_1arg' },
{ oid => '5032', descr => 'list directory with metadata',
proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => 'text bool bool',
- proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
- proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,isdir}',
+ proallargtypes => '{text,bool,bool,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,i,i,o,o,o,o,o,o,o}',
+ proargnames => '{dirname,missing_ok,include_dot_dirs,name,size,access,modification,change,creation,type}',
prosrc => 'pg_ls_dir_metadata' },
{ oid => '5033', descr => 'list directory with metadata',
proname => 'pg_ls_dir_metadata', procost => '10', prorows => '20', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => 'text',
- proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}', proargmodes => '{i,o,o,o,o,o,o,o}',
- proargnames => '{dirname,name,size,access,modification,change,creation,isdir}',
+ proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}', proargmodes => '{i,o,o,o,o,o,o,o}',
+ proargnames => '{dirname,name,size,access,modification,change,creation,type}',
prosrc => 'pg_ls_dir_metadata_1arg' },
{ oid => '5034', descr => 'list all files in a directory recursively',
proname => 'pg_ls_dir_recurse', prorows => '10000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => 'text',
- proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,bool}',
- proargnames => '{dirname,name,size,access,modification,change,creation,isdir}', proargmodes => '{i,o,o,o,o,o,o,o}',
- prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.isdir from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.isdir) select * from ls" },
+ proallargtypes => '{text,text,int8,timestamptz,timestamptz,timestamptz,timestamptz,char}',
+ proargnames => '{dirname,name,size,access,modification,change,creation,type}', proargmodes => '{i,o,o,o,o,o,o,o}',
+ prolang => 'sql', prosrc => "with recursive ls as (select * from pg_ls_dir_metadata(dirname, true, false) union all select ls.name||'/'||a.name, a.size, a.access, a.modification, a.change, a.creation, a.type from ls, lateral pg_ls_dir_metadata(dirname||'/'||ls.name, false, false)a where ls.type='d') select * from ls" },
# hash partitioning constraint function
{ oid => '5028', descr => 'hash partition CHECK constraint',
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out
index cce84a60a9..1bdcca16fc 100644
--- a/src/test/regress/expected/misc_functions.out
+++ b/src/test/regress/expected/misc_functions.out
@@ -157,8 +157,8 @@ select count(*) > 0 as ok from (select pg_ls_waldir()) ss;
-- Test not-run-to-completion cases.
select * from pg_ls_waldir() limit 0;
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
select count(*) > 0 as ok from (select * from pg_ls_waldir() limit 1) ss;
@@ -222,39 +222,39 @@ ERROR: could not open directory "does not exist": No such file or directory
-- This tests the missing_ok parameter, which causes pg_ls_tmpdir to succeed even if the tmpdir doesn't exist yet
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
select * from pg_ls_tmpdir() where name='Does not exist';
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
- name | isdir
-------+-------
- . | t
+select name, type from pg_ls_dir_metadata('.') where name='.';
+ name | type
+------+------
+ . | d
(1 row)
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
- name | isdir
-------+-------
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+ name | type
+------+------
(0 rows)
-- Check that expected columns are present
select * from pg_ls_dir_metadata('.') limit 0;
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
- name | isdir
------------------------+-------
- pg_wal | t
- pg_wal/archive_status | t
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
+ name | type
+-----------------------+------
+ pg_wal | d
+ pg_wal/archive_status | d
(2 rows)
-- Check that expected columns are present
SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
--
diff --git a/src/test/regress/output/tablespace.source b/src/test/regress/output/tablespace.source
index 1e1e02b589..025c9709a1 100644
--- a/src/test/regress/output/tablespace.source
+++ b/src/test/regress/output/tablespace.source
@@ -17,15 +17,15 @@ CREATE TABLESPACE regress_tblspace LOCATION '@testtablespace@';
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
-- The query is written to ERROR if the tablespace doesn't exist, rather than silently failing to call pg_ls_tmpdir()
SELECT c.* FROM (SELECT oid FROM pg_tablespace b WHERE b.spcname='regress_tblspace' UNION SELECT 0 ORDER BY 1 DESC LIMIT 1) AS b , pg_ls_tmpdir(oid) AS c WHERE c.name='Does not exist';
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
-- This tests the missing_ok parameter. If that's not functioning, this would ERROR if the logdir doesn't exist yet.
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
SELECT * FROM pg_ls_logdir() WHERE name='Does not exist';
- name | size | access | modification | change | creation | isdir
-------+------+--------+--------------+--------+----------+-------
+ name | size | access | modification | change | creation | type
+------+------+--------+--------------+--------+----------+------
(0 rows)
-- try setting and resetting some properties for the new tablespace
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sql
index 6041c4f3dc..3874625ca1 100644
--- a/src/test/regress/sql/misc_functions.sql
+++ b/src/test/regress/sql/misc_functions.sql
@@ -69,15 +69,15 @@ select pg_ls_dir('does not exist'); -- fails with missingok=false
-- The name='' condition is never true, so the function runs to completion but returns zero rows.
select * from pg_ls_tmpdir() where name='Does not exist';
-select name, isdir from pg_ls_dir_metadata('.') where name='.';
+select name, type from pg_ls_dir_metadata('.') where name='.';
-select name, isdir from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
+select name, type from pg_ls_dir_metadata('.', false, false) where name='.'; -- include_dot_dirs=false
-- Check that expected columns are present
select * from pg_ls_dir_metadata('.') limit 0;
-- Check that we at least succeed in recursing once, and that we don't show the leading dir prefix
-SELECT name, isdir FROM pg_ls_dir_recurse('.') WHERE isdir AND name~'^pg_wal';
+SELECT name, type FROM pg_ls_dir_recurse('.') WHERE type='d' AND name~'^pg_wal';
-- Check that expected columns are present
SELECT * FROM pg_ls_dir_recurse('.') LIMIT 0;
--
2.17.0
--8w3uRX/HFJGApMzv--
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
@ 2022-08-24 22:47 Zhihong Yu <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Zhihong Yu @ 2022-08-24 22:47 UTC (permalink / raw)
To: PostgreSQL Developers <[email protected]>
Hi,
I was looking at 0004-COPY_IGNORE_ERRORS.patch
+ * Ignore constraints if IGNORE_ERRORS is enabled
+ */
+static void
+safeExecConstraints(CopyFromState cstate, ResultRelInfo *resultRelInfo,
TupleTableSlot *myslot, EState *estate)
I think the existing ExecConstraints() can be expanded by
checking cstate->opts.ignore_errors so that it can selectively
ignore Constraint Violations.
This way you don't need safeExecConstraints().
Cheers
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v1 4/7] Row pattern recognition patch (executor).
@ 2023-06-25 11:48 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-06-25 11:48 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 223 +++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 274 ++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 9 +
src/include/nodes/execnodes.h | 12 ++
src/include/windowapi.h | 9 +
5 files changed, 517 insertions(+), 10 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..ae997dff86 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -48,6 +48,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +160,14 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Map between Var attno in a target list and the parsed attno.
+ */
+typedef struct AttnoMap {
+ List *attno; /* att number in target list (list of AttNumber) */
+ List *attnosyn; /* parsed att number (list of AttNumber) */
+} AttnoMap;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -195,9 +204,9 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static void attno_map(Node *node, AttnoMap *map);
+static bool attno_map_walker(Node *node, void *context);
/*
* initialize_windowaggregate
@@ -2388,6 +2397,12 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+ Var *var;
+ int nargs;
+ AttnoMap attnomap;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2498,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2692,69 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+
+ /*
+ * Collect mapping between varattno and varattnosyn in the targetlist.
+ * XXX: For now we only check RPR's argument. Eventually we have to
+ * recurse the targetlist to find out all mappings in Var nodes.
+ */
+ attnomap.attno = NIL;
+ attnomap.attnosyn = NIL;
+
+ foreach (l, node->plan.targetlist)
+ {
+ te = lfirst(l);
+ if (IsA(te->expr, WindowFunc))
+ {
+ WindowFunc *func = (WindowFunc *)te->expr;
+ if (func->winfnoid != F_RPR)
+ continue;
+
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "RPR must have 1 argument but function %d has %d args", func->winfnoid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "RPR's arg is not Var");
+
+ var = (Var *)expr;
+ elog(DEBUG1, "resname: %s varattno: %d varattnosyn: %d",
+ te->resname, var->varattno, var->varattnosyn);
+ attnomap.attno = lappend_int(attnomap.attno, var->varattno);
+ attnomap.attnosyn = lappend_int(attnomap.attnosyn, var->varattnosyn);
+ }
+ }
+
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ /* tweak expr so that it referes to outer slot */
+ attno_map((Node *)expr, &attnomap);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2762,76 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite Var node's varattno to the varattno which is used in the target
+ * list using AttnoMap. We also rewrite varno so that it sees outer tuple
+ * (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node, AttnoMap *map)
+{
+ (void) expression_tree_walker(node, attno_map_walker, (void *) map);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+ AttnoMap *attnomap;
+ ListCell *lc1, *lc2;
+
+ if (node == NULL)
+ return false;
+
+ attnomap = (AttnoMap *) context;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ var->varno = OUTER_VAR;
+ else
+ var->varno = INNER_VAR;
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+ }
+ else if (IsA(node, Var))
+ {
+ var = (Var *)node;
+
+ elog(DEBUG1, "original varno: %d varattno: %d", var->varno, var->varattno);
+
+ forboth(lc1, attnomap->attno, lc2, attnomap->attnosyn)
+ {
+ int attno = lfirst_int(lc1);
+ int attnosyn = lfirst_int(lc2);
+
+ if (var->varattno == attnosyn)
+ {
+ elog(DEBUG1, "loc: %d rewrite varattno from: %d to %d", var->location, attnosyn, attno);
+ var->varattno = attno;
+ }
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2849,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2900,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3080,7 +3242,7 @@ are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
*
* Returns true if successful, false if no such row
*/
-static bool
+bool
window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
{
WindowAggState *winstate = winobj->winstate;
@@ -3420,14 +3582,53 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3583,15 +3784,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3821,9 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+WindowAggState *
+WinGetAggState(WindowObject winobj)
+{
+ return winobj->winstate;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..63b225271c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -39,7 +42,9 @@ typedef struct
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
+static bool get_slots(WindowObject winobj, WindowAggState *winstate, int current_pos);
+static int evaluate_pattern(WindowObject winobj, WindowAggState *winstate,
+ int relpos, char *vname, char *quantifier, bool *result);
/*
* utility routine for *_rank functions.
@@ -713,3 +718,270 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * rpr
+ * allow to use "Row pattern recognition: WINDOW clause" (SQL:2016 R020) in
+ * the target list.
+ * Usage: SELECT rpr(colname) OVER (..)
+ * where colname is defined in PATTERN clause.
+ */
+Datum
+window_rpr(PG_FUNCTION_ARGS)
+{
+#define MAX_PATTERNS 16 /* max variables in PATTERN clause */
+
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ WindowAggState *winstate = WinGetAggState(winobj);
+ Datum result;
+ bool expression_result;
+ bool isnull;
+ int relpos;
+ int64 curr_pos, markpos;
+ ListCell *lc, *lc1;
+
+ curr_pos = WinGetCurrentPosition(winobj);
+ elog(DEBUG1, "rpr is called. row: " INT64_FORMAT, curr_pos);
+
+ /*
+ * Evaluate PATTERN until one of expressions is not true or out of frame.
+ */
+ relpos = 0;
+
+ forboth(lc, winstate->patternVariableList, lc1, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc));
+ char *quantifier = strVal(lfirst(lc1));
+
+ elog(DEBUG1, "relpos: %d pattern vname: %s quantifier: %s", relpos, vname, quantifier);
+
+ /* evaluate row pattern against current row */
+ relpos = evaluate_pattern(winobj, winstate, relpos, vname, quantifier, &expression_result);
+
+ /*
+ * If the expression did not match, we are done.
+ */
+ if (!expression_result)
+ break;
+
+ /* out of frame? */
+ if (relpos < 0)
+ break;
+
+ /* count up relative row position */
+ relpos++;
+ }
+
+ elog(DEBUG1, "relpos: %d", relpos);
+
+ /*
+ * If current row satified the pattern, return argument expression.
+ */
+ if (expression_result)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ 0, WINDOW_SEEK_HEAD, false,
+ &isnull, NULL);
+ }
+
+ /*
+ * At this point we can set mark down to current pos -2.
+ */
+ markpos = curr_pos -2;
+ elog(DEBUG1, "markpos: " INT64_FORMAT, markpos);
+ if (markpos >= 0)
+ WinSetMarkPosition(winobj, markpos);
+
+ if (expression_result)
+ PG_RETURN_DATUM(result);
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match relative row position
+ * -1: current row is out of frame
+ */
+static
+int evaluate_pattern(WindowObject winobj, WindowAggState *winstate,
+ int relpos, char *vname, char *quantifier, bool *result)
+{
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2;
+ ExprState *pat;
+ Datum eval_result;
+ int sts;
+ bool out_of_frame = false;
+ bool isnull;
+ StringInfo encoded_str = makeStringInfo();
+ char pattern_str[128];
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList)
+ {
+ char *name = strVal(lfirst(lc1));
+ bool second_try_match = false;
+
+ if (strcmp(vname, name))
+ continue;
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ for (;;)
+ {
+ if (!get_slots(winobj, winstate, relpos))
+ {
+ out_of_frame = true;
+ break; /* current row is out of frame */
+ }
+
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: %d", vname, relpos);
+ break;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: %d", vname, relpos);
+ break;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: %d", vname, relpos);
+ appendStringInfoChar(encoded_str, vname[0]);
+
+ /* If quantifier is "+", we need to look for more matching row */
+ if (quantifier && !strcmp(quantifier, "+"))
+ {
+ /* remember that we want to try another row */
+ second_try_match = true;
+ relpos++;
+ }
+ else
+ break;
+ }
+ }
+ }
+ if (second_try_match)
+ relpos--;
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+
+ /* build regular expression */
+ snprintf(pattern_str, sizeof(pattern_str), "%c%s", vname[0], quantifier);
+
+ /*
+ * Do regular expression matching against sequence of rows satisfying
+ * the expression using regexp_instr().
+ */
+ sts = DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str->data)),
+ PointerGetDatum(cstring_to_text(pattern_str))));
+ elog(DEBUG1, "regexp_instr returned: %d. str: %s regexp: %s",
+ sts, encoded_str->data, pattern_str);
+ *result = (sts > 0)? true : false;
+ }
+ return relpos;
+}
+
+/*
+ * Get current, previous and next tuple.
+ * Returns true if still within frame.
+ */
+static bool
+get_slots(WindowObject winobj, WindowAggState *winstate, int current_pos)
+{
+ TupleTableSlot *slot;
+ bool isnull, isout;
+ int sts;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* for current row */
+ slot = winstate->temp_slot_1;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "current row is out of frame");
+ econtext->ecxt_scantuple = winstate->null_slot;
+ return false;
+ }
+ else
+ econtext->ecxt_scantuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos - 1, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "previous row out of frame at: %d", current_pos);
+ econtext->ecxt_outertuple = winstate->null_slot;
+ }
+ econtext->ecxt_outertuple = slot;
+ }
+ else
+ econtext->ecxt_outertuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos + 1, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "next row out of frame at: %d", current_pos);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+
+ return true;
+}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..e3a9e0ffeb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10397,6 +10397,15 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'row pattern recognition in window',
+ proname => 'rpr', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_rpr' },
+{ oid => '6123', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6124', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..bc95c5fe9b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2519,6 +2519,13 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('*', '+' or '?'. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2562,11 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
} WindowAggState;
/* ----------------
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index b8c2c565d1..a0facf38fe 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -58,7 +58,16 @@ extern Datum WinGetFuncArgInFrame(WindowObject winobj, int argno,
int relpos, int seektype, bool set_mark,
bool *isnull, bool *isout);
+extern int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+
extern Datum WinGetFuncArgCurrent(WindowObject winobj, int argno,
bool *isnull);
+extern WindowAggState *WinGetAggState(WindowObject winobj);
+
+extern bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+
#endif /* WINDOWAPI_H */
--
2.25.1
----Next_Part(Sun_Jun_25_21_05_09_2023_126)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v1-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v2 4/7] Row pattern recognition patch (executor).
@ 2023-06-26 08:05 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-06-26 08:05 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 225 +++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 302 ++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 9 +
src/include/nodes/execnodes.h | 13 ++
src/include/windowapi.h | 9 +
5 files changed, 548 insertions(+), 10 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..bef2bc62b2 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -48,6 +48,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +160,14 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Map between Var attno in a target list and the parsed attno.
+ */
+typedef struct AttnoMap {
+ List *attno; /* att number in target list (list of AttNumber) */
+ List *attnosyn; /* parsed att number (list of AttNumber) */
+} AttnoMap;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -195,9 +204,9 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static void attno_map(Node *node, AttnoMap *map);
+static bool attno_map_walker(Node *node, void *context);
/*
* initialize_windowaggregate
@@ -2388,6 +2397,12 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+ Var *var;
+ int nargs;
+ AttnoMap attnomap;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2498,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2692,71 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+
+ /*
+ * Collect mapping between varattno and varattnosyn in the targetlist.
+ * XXX: For now we only check RPR's argument. Eventually we have to
+ * recurse the targetlist to find out all mappings in Var nodes.
+ */
+ attnomap.attno = NIL;
+ attnomap.attnosyn = NIL;
+
+ foreach (l, node->plan.targetlist)
+ {
+ te = lfirst(l);
+ if (IsA(te->expr, WindowFunc))
+ {
+ WindowFunc *func = (WindowFunc *)te->expr;
+ if (func->winfnoid != F_RPR)
+ continue;
+
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "RPR must have 1 argument but function %d has %d args", func->winfnoid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "RPR's arg is not Var");
+
+ var = (Var *)expr;
+ elog(DEBUG1, "resname: %s varattno: %d varattnosyn: %d",
+ te->resname, var->varattno, var->varattnosyn);
+ attnomap.attno = lappend_int(attnomap.attno, var->varattno);
+ attnomap.attnosyn = lappend_int(attnomap.attnosyn, var->varattnosyn);
+ }
+ }
+
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ /* tweak expr so that it referes to outer slot */
+ attno_map((Node *)expr, &attnomap);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2764,76 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite Var node's varattno to the varattno which is used in the target
+ * list using AttnoMap. We also rewrite varno so that it sees outer tuple
+ * (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node, AttnoMap *map)
+{
+ (void) expression_tree_walker(node, attno_map_walker, (void *) map);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+ AttnoMap *attnomap;
+ ListCell *lc1, *lc2;
+
+ if (node == NULL)
+ return false;
+
+ attnomap = (AttnoMap *) context;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ var->varno = OUTER_VAR;
+ else
+ var->varno = INNER_VAR;
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+ }
+ else if (IsA(node, Var))
+ {
+ var = (Var *)node;
+
+ elog(DEBUG1, "original varno: %d varattno: %d", var->varno, var->varattno);
+
+ forboth(lc1, attnomap->attno, lc2, attnomap->attnosyn)
+ {
+ int attno = lfirst_int(lc1);
+ int attnosyn = lfirst_int(lc2);
+
+ if (var->varattno == attnosyn)
+ {
+ elog(DEBUG1, "loc: %d rewrite varattno from: %d to %d", var->location, attnosyn, attno);
+ var->varattno = attno;
+ }
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2851,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2902,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3080,7 +3244,7 @@ are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
*
* Returns true if successful, false if no such row
*/
-static bool
+bool
window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
{
WindowAggState *winstate = winobj->winstate;
@@ -3420,14 +3584,53 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3583,15 +3786,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3823,9 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+WindowAggState *
+WinGetAggState(WindowObject winobj)
+{
+ return winobj->winstate;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..74ef11ce55 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,10 +39,21 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
+static bool get_slots(WindowObject winobj, WindowAggState *winstate, int current_pos);
+static int evaluate_pattern(WindowObject winobj, WindowAggState *winstate,
+ int relpos, char *vname, char *quantifier, bool *result);
/*
* utility routine for *_rank functions.
@@ -713,3 +727,289 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * rpr
+ * allow to use "Row pattern recognition: WINDOW clause" (SQL:2016 R020) in
+ * the target list.
+ * Usage: SELECT rpr(colname) OVER (..)
+ * where colname is defined in PATTERN clause.
+ */
+Datum
+window_rpr(PG_FUNCTION_ARGS)
+{
+#define MAX_PATTERNS 16 /* max variables in PATTERN clause */
+
+ WindowObject winobj = PG_WINDOW_OBJECT();
+ WindowAggState *winstate = WinGetAggState(winobj);
+ Datum result;
+ bool expression_result;
+ bool isnull;
+ int relpos;
+ int64 curr_pos, markpos;
+ ListCell *lc, *lc1;
+ SkipContext *context = NULL;
+
+ curr_pos = WinGetCurrentPosition(winobj);
+ elog(DEBUG1, "rpr is called. row: " INT64_FORMAT, curr_pos);
+
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW)
+ {
+ context = (SkipContext *) WinGetPartitionLocalMemory(winobj, sizeof(SkipContext));
+ if (curr_pos < context->pos)
+ {
+ elog(DEBUG1, "skip this row: curr_pos: " INT64_FORMAT "context->pos: " INT64_FORMAT,
+ curr_pos, context->pos);
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Evaluate PATTERN until one of expressions is not true or out of frame.
+ */
+ relpos = 0;
+
+ forboth(lc, winstate->patternVariableList, lc1, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc));
+ char *quantifier = strVal(lfirst(lc1));
+
+ elog(DEBUG1, "relpos: %d pattern vname: %s quantifier: %s", relpos, vname, quantifier);
+
+ /* evaluate row pattern against current row */
+ relpos = evaluate_pattern(winobj, winstate, relpos, vname, quantifier, &expression_result);
+
+ /*
+ * If the expression did not match, we are done.
+ */
+ if (!expression_result)
+ break;
+
+ /* out of frame? */
+ if (relpos < 0)
+ break;
+
+ /* count up relative row position */
+ relpos++;
+ }
+
+ elog(DEBUG1, "relpos: %d", relpos);
+
+ /*
+ * If current row satified the pattern, return argument expression.
+ */
+ if (expression_result)
+ {
+ result = WinGetFuncArgInFrame(winobj, 0,
+ 0, WINDOW_SEEK_HEAD, false,
+ &isnull, NULL);
+ }
+
+ /*
+ * At this point we can set mark down to current pos -2.
+ */
+ markpos = curr_pos -2;
+ elog(DEBUG1, "markpos: " INT64_FORMAT, markpos);
+ if (markpos >= 0)
+ WinSetMarkPosition(winobj, markpos);
+
+ if (expression_result)
+ {
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW)
+ {
+ context->pos += relpos;
+ elog(DEBUG1, "context->pos: " INT64_FORMAT, context->pos);
+ }
+ PG_RETURN_DATUM(result);
+ }
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match relative row position
+ * -1: current row is out of frame
+ */
+static
+int evaluate_pattern(WindowObject winobj, WindowAggState *winstate,
+ int relpos, char *vname, char *quantifier, bool *result)
+{
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2;
+ ExprState *pat;
+ Datum eval_result;
+ int sts;
+ bool out_of_frame = false;
+ bool isnull;
+ StringInfo encoded_str = makeStringInfo();
+ char pattern_str[128];
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList)
+ {
+ char *name = strVal(lfirst(lc1));
+ bool second_try_match = false;
+
+ if (strcmp(vname, name))
+ continue;
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ for (;;)
+ {
+ if (!get_slots(winobj, winstate, relpos))
+ {
+ out_of_frame = true;
+ break; /* current row is out of frame */
+ }
+
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: %d", vname, relpos);
+ break;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: %d", vname, relpos);
+ break;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: %d", vname, relpos);
+ appendStringInfoChar(encoded_str, vname[0]);
+
+ /* If quantifier is "+", we need to look for more matching row */
+ if (quantifier && !strcmp(quantifier, "+"))
+ {
+ /* remember that we want to try another row */
+ second_try_match = true;
+ relpos++;
+ }
+ else
+ break;
+ }
+ }
+ }
+ if (second_try_match)
+ relpos--;
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+
+ /* build regular expression */
+ snprintf(pattern_str, sizeof(pattern_str), "%c%s", vname[0], quantifier);
+
+ /*
+ * Do regular expression matching against sequence of rows satisfying
+ * the expression using regexp_instr().
+ */
+ sts = DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str->data)),
+ PointerGetDatum(cstring_to_text(pattern_str))));
+ elog(DEBUG1, "regexp_instr returned: %d. str: %s regexp: %s",
+ sts, encoded_str->data, pattern_str);
+ *result = (sts > 0)? true : false;
+ }
+ return relpos;
+}
+
+/*
+ * Get current, previous and next tuple.
+ * Returns true if still within frame.
+ */
+static bool
+get_slots(WindowObject winobj, WindowAggState *winstate, int current_pos)
+{
+ TupleTableSlot *slot;
+ bool isnull, isout;
+ int sts;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* for current row */
+ slot = winstate->temp_slot_1;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "current row is out of frame");
+ econtext->ecxt_scantuple = winstate->null_slot;
+ return false;
+ }
+ else
+ econtext->ecxt_scantuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos - 1, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "previous row out of frame at: %d", current_pos);
+ econtext->ecxt_outertuple = winstate->null_slot;
+ }
+ econtext->ecxt_outertuple = slot;
+ }
+ else
+ econtext->ecxt_outertuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ sts = WinGetSlotInFrame(winobj, slot,
+ current_pos + 1, WINDOW_SEEK_HEAD, false,
+ &isnull, &isout);
+ if (sts < 0)
+ {
+ elog(DEBUG1, "next row out of frame at: %d", current_pos);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+
+ return true;
+}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..e3a9e0ffeb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10397,6 +10397,15 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'row pattern recognition in window',
+ proname => 'rpr', prokind => 'w', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_rpr' },
+{ oid => '6123', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6124', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..1643eaa6f1 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2519,6 +2519,14 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2563,11 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
} WindowAggState;
/* ----------------
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index b8c2c565d1..a0facf38fe 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -58,7 +58,16 @@ extern Datum WinGetFuncArgInFrame(WindowObject winobj, int argno,
int relpos, int seektype, bool set_mark,
bool *isnull, bool *isout);
+extern int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+
extern Datum WinGetFuncArgCurrent(WindowObject winobj, int argno,
bool *isnull);
+extern WindowAggState *WinGetAggState(WindowObject winobj);
+
+extern bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+
#endif /* WINDOWAPI_H */
--
2.25.1
----Next_Part(Mon_Jun_26_17_45_07_2023_724)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v2-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v3 4/7] Row pattern recognition patch (executor).
@ 2023-07-26 10:49 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-07-26 10:49 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 701 ++++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 38 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 18 +
src/include/windowapi.h | 8 +
5 files changed, 758 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..0586bf57d6 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,14 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Map between Var attno in a target list and the parsed attno.
+ */
+typedef struct AttnoMap {
+ List *attno; /* att number in target list (list of AttNumber) */
+ List *attnosyn; /* parsed att number (list of AttNumber) */
+} AttnoMap;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +192,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +206,19 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static void attno_map(Node *node, AttnoMap *map);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringInfo *str_set, int set_size);
+static void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index,
+ char *encoded_str, int *resultlen);
/*
* initialize_windowaggregate
@@ -673,6 +694,9 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool reduced_frame_set;
+ bool check_reduced_frame;
+ int num_rows_in_reduced_frame;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -790,6 +814,7 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
winstate->aggregatedupto <= winstate->frameheadpos)
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +886,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -919,6 +946,10 @@ eval_windowaggregates(WindowAggState *winstate)
ExecClearTuple(agg_row_slot);
}
+ reduced_frame_set = false;
+ check_reduced_frame = false;
+ num_rows_in_reduced_frame = 0;
+
/*
* Advance until we reach a row not in frame (or end of partition).
*
@@ -930,12 +961,18 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
if (!window_gettupleslot(agg_winobj, winstate->aggregatedupto,
agg_row_slot))
+ {
+ if (check_reduced_frame)
+ winstate->aggregatedupto--;
break; /* must be end of partition */
+ }
}
/*
@@ -944,10 +981,47 @@ eval_windowaggregates(WindowAggState *winstate)
*/
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
+ {
+ if (winstate->patternVariableList != NIL && check_reduced_frame)
+ winstate->aggregatedupto--;
break;
+ }
if (ret == 0)
goto next_tuple;
+ if (winstate->patternVariableList != NIL)
+ {
+ if (!reduced_frame_set)
+ {
+ num_rows_in_reduced_frame = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ reduced_frame_set = true;
+ elog(DEBUG1, "set num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+
+ if (num_rows_in_reduced_frame <= 0)
+ break;
+
+ else if (num_rows_in_reduced_frame > 0)
+ check_reduced_frame = true;
+ }
+
+ if (check_reduced_frame)
+ {
+ elog(DEBUG1, "decrease num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+ num_rows_in_reduced_frame--;
+ if (num_rows_in_reduced_frame < 0)
+ {
+ /*
+ * No more rows remain in the reduced frame. Finish
+ * accumulating row into the aggregates.
+ */
+ winstate->aggregatedupto--;
+ break;
+ }
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1050,8 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+ elog(DEBUG1, "===== break loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -2053,6 +2129,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2388,6 +2466,12 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+ Var *var;
+ int nargs;
+ AttnoMap attnomap;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2567,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2761,69 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+
+ /*
+ * Collect mapping between varattno and varattnosyn in the targetlist.
+ * XXX: For now we only check RPR's argument. Eventually we have to
+ * recurse the targetlist to find out all mappings in Var nodes.
+ */
+ attnomap.attno = NIL;
+ attnomap.attnosyn = NIL;
+
+ foreach (l, node->plan.targetlist)
+ {
+ te = lfirst(l);
+ if (IsA(te->expr, WindowFunc))
+ {
+ WindowFunc *func = (WindowFunc *)te->expr;
+
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (nargs != 1)
+ continue;
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ continue;
+
+ var = (Var *)expr;
+ elog(DEBUG1, "resname: %s varattno: %d varattnosyn: %d",
+ te->resname, var->varattno, var->varattnosyn);
+ attnomap.attno = lappend_int(attnomap.attno, var->varattno);
+ attnomap.attnosyn = lappend_int(attnomap.attnosyn, var->varattnosyn);
+ }
+ }
+
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ /* tweak expr so that it referes to outer slot */
+ attno_map((Node *)expr, &attnomap);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2831,77 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite Var node's varattno to the varattno which is used in the target
+ * list using AttnoMap. We also rewrite varno so that it sees outer tuple
+ * (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node, AttnoMap *map)
+{
+ (void) expression_tree_walker(node, attno_map_walker, (void *) map);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+ AttnoMap *attnomap;
+ ListCell *lc1, *lc2;
+
+ if (node == NULL)
+ return false;
+
+ attnomap = (AttnoMap *) context;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ var->varno = OUTER_VAR;
+ else
+ var->varno = INNER_VAR;
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+ }
+ else if (IsA(node, Var))
+ {
+ var = (Var *)node;
+
+ elog(DEBUG1, "original varno: %d varattno: %d", var->varno, var->varattno);
+
+ forboth(lc1, attnomap->attno, lc2, attnomap->attnosyn)
+ {
+ int attno = lfirst_int(lc1);
+ int attnosyn = lfirst_int(lc2);
+
+ elog(DEBUG1, "walker: varattno: %d varattnosyn: %d",attno, attnosyn);
+ if (var->varattno == attnosyn)
+ {
+ elog(DEBUG1, "loc: %d rewrite varattno from: %d to %d", var->location, attnosyn, attno);
+ var->varattno = attno;
+ }
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, (void *) context);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2919,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2970,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3080,7 +3312,7 @@ are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
*
* Returns true if successful, false if no such row
*/
-static bool
+bool
window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
{
WindowAggState *winstate = winobj->winstate;
@@ -3100,7 +3332,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3652,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,6 +3766,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
@@ -3565,6 +3843,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3867,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3904,400 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+WindowAggState *
+WinGetAggState(WindowObject winobj)
+{
+ return winobj->winstate;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+
+ /*
+ * Array of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+ #define ENCODED_STR_ARRAY_ALLOC_SIZE 128
+ StringInfo *str_set = NULL;
+ int str_set_index;
+ int str_set_size;
+
+ if (winstate->patternVariableList == NIL)
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ return 0;
+ }
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check whether the row speicied by pos is in the reduced frame. The
+ * second and subsequent rows need to be recognized as "unmatched" rows if
+ * AFTER MATCH SKIP PAST LAST ROW is defined.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ pos > winstate->headpos_in_reduced_frame &&
+ pos < (winstate->headpos_in_reduced_frame + winstate->num_rows_in_reduced_frame))
+ return -2;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ /* build encoded string array */
+ if (str_set == NULL)
+ {
+ str_set_index = 0;
+ str_set_size = ENCODED_STR_ARRAY_ALLOC_SIZE * sizeof(StringInfo);
+ str_set = palloc(str_set_size);
+ }
+
+ str_set[str_set_index++] = encoded_str;
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ if (str_set_index >= str_set_size)
+ {
+ str_set_size *= 2;
+ str_set = repalloc(str_set, str_set_size);
+ }
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (str_set == NULL)
+ {
+ /* no matches found in the first row */
+ return -1;
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ appendStringInfoChar(pattern_str, vname[0]);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s quantifier: %s", vname, quantifier);
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set, str_set_index);
+ if (num_matched_rows <= 0)
+ return -1;
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ winstate->headpos_in_reduced_frame = original_pos;
+ winstate->num_rows_in_reduced_frame = num_matched_rows;
+
+ return num_matched_rows;
+}
+
+/*
+ * search set of encode_str.
+ * set_size: size of set_str array.
+ */
+static
+int search_str_set(char *pattern, StringInfo *str_set, int set_size)
+{
+ char *encoded_str = palloc0(set_size+1);
+ int resultlen = 0;
+
+ search_str_set_recurse(pattern, str_set, set_size, 0, encoded_str, &resultlen);
+ elog(DEBUG1, "search_str_set returns %d", resultlen);
+ return resultlen;
+}
+
+static
+void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index, char *encoded_str, int *resultlen)
+{
+ char *p;
+
+ if (set_index >= set_size)
+ {
+ Datum d;
+ text *res;
+ char *substr;
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how log the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr is, it error
+ * out if pattern is not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ int len;
+
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ if (len > *resultlen)
+ /* remember the longest match */
+ *resultlen = len;
+ }
+ }
+ return;
+ }
+
+ p = str_set[set_index]->data;
+ while (*p)
+ {
+ encoded_str[set_index] = *p;
+ p++;
+ search_str_set_recurse(pattern, str_set, set_size, set_index + 1, encoded_str, resultlen);
+ }
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList)
+ {
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, vname[0]);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_scantuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_outertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_outertuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_outertuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_outertuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..e4cab36ec9 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,26 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..fa100b2665 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10397,6 +10397,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..4fd3bd1a93 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2519,6 +2519,14 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2563,16 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /* head of the reduced window frame */
+ int64 headpos_in_reduced_frame;
+ /* number of rows in the reduced window frame */
+ int64 num_rows_in_reduced_frame;
} WindowAggState;
/* ----------------
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index b8c2c565d1..1e292648e9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -58,7 +58,15 @@ extern Datum WinGetFuncArgInFrame(WindowObject winobj, int argno,
int relpos, int seektype, bool set_mark,
bool *isnull, bool *isout);
+extern int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+
extern Datum WinGetFuncArgCurrent(WindowObject winobj, int argno,
bool *isnull);
+extern WindowAggState *WinGetAggState(WindowObject winobj);
+
+extern bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
#endif /* WINDOWAPI_H */
--
2.25.1
----Next_Part(Wed_Jul_26_21_21_34_2023_317)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v3-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v4 4/7] Row pattern recognition patch (executor).
@ 2023-08-09 07:56 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 674 ++++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 38 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 19 +
src/include/windowapi.h | 8 +
5 files changed, 732 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..5354e47045 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -182,8 +184,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +198,20 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringInfo *str_set, int set_size);
+static void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index,
+ char *encoded_str, int *resultlen);
+static char pattern_initial(WindowAggState *winstate, char *vname);
/*
* initialize_windowaggregate
@@ -673,6 +687,9 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool reduced_frame_set;
+ bool check_reduced_frame;
+ int num_rows_in_reduced_frame;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -790,6 +807,7 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
winstate->aggregatedupto <= winstate->frameheadpos)
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +879,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -919,6 +939,10 @@ eval_windowaggregates(WindowAggState *winstate)
ExecClearTuple(agg_row_slot);
}
+ reduced_frame_set = false;
+ check_reduced_frame = false;
+ num_rows_in_reduced_frame = 0;
+
/*
* Advance until we reach a row not in frame (or end of partition).
*
@@ -930,12 +954,18 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
if (!window_gettupleslot(agg_winobj, winstate->aggregatedupto,
agg_row_slot))
+ {
+ if (check_reduced_frame)
+ winstate->aggregatedupto--;
break; /* must be end of partition */
+ }
}
/*
@@ -944,10 +974,47 @@ eval_windowaggregates(WindowAggState *winstate)
*/
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
+ {
+ if (winstate->patternVariableList != NIL && check_reduced_frame)
+ winstate->aggregatedupto--;
break;
+ }
if (ret == 0)
goto next_tuple;
+ if (winstate->patternVariableList != NIL)
+ {
+ if (!reduced_frame_set)
+ {
+ num_rows_in_reduced_frame = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ reduced_frame_set = true;
+ elog(DEBUG1, "set num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+
+ if (num_rows_in_reduced_frame <= 0)
+ break;
+
+ else if (num_rows_in_reduced_frame > 0)
+ check_reduced_frame = true;
+ }
+
+ if (check_reduced_frame)
+ {
+ elog(DEBUG1, "decrease num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+ num_rows_in_reduced_frame--;
+ if (num_rows_in_reduced_frame < 0)
+ {
+ /*
+ * No more rows remain in the reduced frame. Finish
+ * accumulating row into the aggregates.
+ */
+ winstate->aggregatedupto--;
+ break;
+ }
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1043,8 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+ elog(DEBUG1, "===== break loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -2053,6 +2122,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2388,6 +2459,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2557,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2751,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2791,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2859,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2910,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3080,7 +3252,7 @@ are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
*
* Returns true if successful, false if no such row
*/
-static bool
+bool
window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
{
WindowAggState *winstate = winobj->winstate;
@@ -3100,7 +3272,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3592,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,6 +3706,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
@@ -3565,6 +3783,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3807,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3844,433 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+WindowAggState *
+WinGetAggState(WindowObject winobj)
+{
+ return winobj->winstate;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+
+ /*
+ * Array of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+ #define ENCODED_STR_ARRAY_ALLOC_SIZE 128
+ StringInfo *str_set = NULL;
+ int str_set_index;
+ int str_set_size;
+
+ if (winstate->patternVariableList == NIL)
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ return 0;
+ }
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check whether the row speicied by pos is in the reduced frame. The
+ * second and subsequent rows need to be recognized as "unmatched" rows if
+ * AFTER MATCH SKIP PAST LAST ROW is defined.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ pos > winstate->headpos_in_reduced_frame &&
+ pos < (winstate->headpos_in_reduced_frame + winstate->num_rows_in_reduced_frame))
+ return -2;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ /* build encoded string array */
+ if (str_set == NULL)
+ {
+ str_set_index = 0;
+ str_set_size = ENCODED_STR_ARRAY_ALLOC_SIZE * sizeof(StringInfo);
+ str_set = palloc(str_set_size);
+ }
+
+ str_set[str_set_index++] = encoded_str;
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ if (str_set_index >= str_set_size)
+ {
+ str_set_size *= 2;
+ str_set = repalloc(str_set, str_set_size);
+ }
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (str_set == NULL)
+ {
+ /* no matches found in the first row */
+ return -1;
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set, str_set_index);
+ if (num_matched_rows <= 0)
+ return -1;
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ winstate->headpos_in_reduced_frame = original_pos;
+ winstate->num_rows_in_reduced_frame = num_matched_rows;
+
+ return num_matched_rows;
+}
+
+/*
+ * search set of encode_str.
+ * set_size: size of set_str array.
+ */
+static
+int search_str_set(char *pattern, StringInfo *str_set, int set_size)
+{
+ char *encoded_str = palloc0(set_size+1);
+ int resultlen = 0;
+
+ search_str_set_recurse(pattern, str_set, set_size, 0, encoded_str, &resultlen);
+ elog(DEBUG1, "search_str_set returns %d", resultlen);
+ return resultlen;
+}
+
+static
+void search_str_set_recurse(char *pattern, StringInfo *str_set,
+ int set_size, int set_index, char *encoded_str, int *resultlen)
+{
+ char *p;
+
+ if (set_index >= set_size)
+ {
+ Datum d;
+ text *res;
+ char *substr;
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how log the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr is, it error
+ * out if pattern is not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ int len;
+
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ if (len > *resultlen)
+ /* remember the longest match */
+ *resultlen = len;
+ }
+ }
+ return;
+ }
+
+ p = str_set[set_index]->data;
+ while (*p)
+ {
+ encoded_str[set_index] = *p;
+ p++;
+ search_str_set_recurse(pattern, str_set, set_size, set_index + 1, encoded_str, resultlen);
+ }
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ elog(DEBUG1, "evaluate_pattern: define variable: %s, pattern variable: %s", name, vname);
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..e4cab36ec9 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,26 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6996073989..fa100b2665 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10397,6 +10397,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..2bd6fcb5e1 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2519,6 +2519,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2564,16 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /* head of the reduced window frame */
+ int64 headpos_in_reduced_frame;
+ /* number of rows in the reduced window frame */
+ int64 num_rows_in_reduced_frame;
} WindowAggState;
/* ----------------
diff --git a/src/include/windowapi.h b/src/include/windowapi.h
index b8c2c565d1..1e292648e9 100644
--- a/src/include/windowapi.h
+++ b/src/include/windowapi.h
@@ -58,7 +58,15 @@ extern Datum WinGetFuncArgInFrame(WindowObject winobj, int argno,
int relpos, int seektype, bool set_mark,
bool *isnull, bool *isout);
+extern int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+
extern Datum WinGetFuncArgCurrent(WindowObject winobj, int argno,
bool *isnull);
+extern WindowAggState *WinGetAggState(WindowObject winobj);
+
+extern bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
#endif /* WINDOWAPI_H */
--
2.25.1
----Next_Part(Wed_Aug__9_17_41_12_2023_134)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v4-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v5 4/7] Row pattern recognition patch (executor).
@ 2023-09-02 06:32 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-09-02 06:32 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 671 ++++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 38 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 19 +
4 files changed, 722 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..2e59369a71 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -182,8 +184,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +198,25 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringInfo *str_set, int set_size);
+static void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index,
+ char *encoded_str, int *resultlen);
+static char pattern_initial(WindowAggState *winstate, char *vname);
/*
* initialize_windowaggregate
@@ -673,6 +692,9 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool reduced_frame_set;
+ bool check_reduced_frame;
+ int num_rows_in_reduced_frame;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -790,6 +812,7 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
winstate->aggregatedupto <= winstate->frameheadpos)
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +884,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -919,6 +944,10 @@ eval_windowaggregates(WindowAggState *winstate)
ExecClearTuple(agg_row_slot);
}
+ reduced_frame_set = false;
+ check_reduced_frame = false;
+ num_rows_in_reduced_frame = 0;
+
/*
* Advance until we reach a row not in frame (or end of partition).
*
@@ -930,12 +959,18 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
if (!window_gettupleslot(agg_winobj, winstate->aggregatedupto,
agg_row_slot))
+ {
+ if (check_reduced_frame)
+ winstate->aggregatedupto--;
break; /* must be end of partition */
+ }
}
/*
@@ -944,10 +979,47 @@ eval_windowaggregates(WindowAggState *winstate)
*/
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
+ {
+ if (winstate->patternVariableList != NIL && check_reduced_frame)
+ winstate->aggregatedupto--;
break;
+ }
if (ret == 0)
goto next_tuple;
+ if (winstate->patternVariableList != NIL)
+ {
+ if (!reduced_frame_set)
+ {
+ num_rows_in_reduced_frame = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ reduced_frame_set = true;
+ elog(DEBUG1, "set num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+
+ if (num_rows_in_reduced_frame <= 0)
+ break;
+
+ else if (num_rows_in_reduced_frame > 0)
+ check_reduced_frame = true;
+ }
+
+ if (check_reduced_frame)
+ {
+ elog(DEBUG1, "decrease num_rows_in_reduced_frame: %d pos: " INT64_FORMAT,
+ num_rows_in_reduced_frame, winstate->aggregatedupto);
+ num_rows_in_reduced_frame--;
+ if (num_rows_in_reduced_frame < 0)
+ {
+ /*
+ * No more rows remain in the reduced frame. Finish
+ * accumulating row into the aggregates.
+ */
+ winstate->aggregatedupto--;
+ break;
+ }
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1048,8 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+ elog(DEBUG1, "===== break loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -2053,6 +2127,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2388,6 +2464,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2562,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2756,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2796,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2864,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2915,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3100,7 +3277,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3597,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,6 +3711,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
@@ -3565,6 +3788,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3812,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3849,427 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+
+ /*
+ * Array of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+ #define ENCODED_STR_ARRAY_ALLOC_SIZE 128
+ StringInfo *str_set = NULL;
+ int str_set_index;
+ int str_set_size;
+
+ if (winstate->patternVariableList == NIL)
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ return 0;
+ }
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check whether the row speicied by pos is in the reduced frame. The
+ * second and subsequent rows need to be recognized as "unmatched" rows if
+ * AFTER MATCH SKIP PAST LAST ROW is defined.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ pos > winstate->headpos_in_reduced_frame &&
+ pos < (winstate->headpos_in_reduced_frame + winstate->num_rows_in_reduced_frame))
+ return -2;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ /* build encoded string array */
+ if (str_set == NULL)
+ {
+ str_set_index = 0;
+ str_set_size = ENCODED_STR_ARRAY_ALLOC_SIZE * sizeof(StringInfo);
+ str_set = palloc(str_set_size);
+ }
+
+ str_set[str_set_index++] = encoded_str;
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ if (str_set_index >= str_set_size)
+ {
+ str_set_size *= 2;
+ str_set = repalloc(str_set, str_set_size);
+ }
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (str_set == NULL)
+ {
+ /* no matches found in the first row */
+ return -1;
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set, str_set_index);
+ if (num_matched_rows <= 0)
+ return -1;
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ winstate->headpos_in_reduced_frame = original_pos;
+ winstate->num_rows_in_reduced_frame = num_matched_rows;
+
+ return num_matched_rows;
+}
+
+/*
+ * search set of encode_str.
+ * set_size: size of set_str array.
+ */
+static
+int search_str_set(char *pattern, StringInfo *str_set, int set_size)
+{
+ char *encoded_str = palloc0(set_size+1);
+ int resultlen = 0;
+
+ search_str_set_recurse(pattern, str_set, set_size, 0, encoded_str, &resultlen);
+ elog(DEBUG1, "search_str_set returns %d", resultlen);
+ return resultlen;
+}
+
+static
+void search_str_set_recurse(char *pattern, StringInfo *str_set,
+ int set_size, int set_index, char *encoded_str, int *resultlen)
+{
+ char *p;
+
+ if (set_index >= set_size)
+ {
+ Datum d;
+ text *res;
+ char *substr;
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how log the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr is, it error
+ * out if pattern is not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ int len;
+
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ if (len > *resultlen)
+ /* remember the longest match */
+ *resultlen = len;
+ }
+ }
+ return;
+ }
+
+ p = str_set[set_index]->data;
+ while (*p)
+ {
+ encoded_str[set_index] = *p;
+ p++;
+ search_str_set_recurse(pattern, str_set, set_size, set_index + 1, encoded_str, resultlen);
+ }
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ elog(DEBUG1, "evaluate_pattern: define variable: %s, pattern variable: %s", name, vname);
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..e4cab36ec9 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,26 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..d20f803cf5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..2bd6fcb5e1 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2519,6 +2519,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2564,16 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /* head of the reduced window frame */
+ int64 headpos_in_reduced_frame;
+ /* number of rows in the reduced window frame */
+ int64 num_rows_in_reduced_frame;
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Sat_Sep__2_15_52_35_2023_273)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v5-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v6 4/7] Row pattern recognition patch (executor).
@ 2023-09-12 05:22 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-09-12 05:22 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 842 ++++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 898 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..32270d051a 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -182,8 +184,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +198,32 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringInfo *str_set, int set_size);
+static void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index,
+ char *encoded_str, int *resultlen);
+static char pattern_initial(WindowAggState *winstate, char *vname);
/*
* initialize_windowaggregate
@@ -673,6 +699,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +805,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +818,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +894,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +952,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row or an unmatched row, we don't need to
+ * accumulate rows, just return NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ (get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED ||
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_UNMATCHED))
+ agg_result_isnull = true;
}
/*
@@ -930,6 +988,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1008,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1058,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1079,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is enabled and we just return NULL. because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row or unmatched row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1183,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2147,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2317,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2495,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2593,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2787,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2827,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2895,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2946,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3100,7 +3308,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3628,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,11 +3742,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3565,6 +3823,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3847,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3884,561 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+
+ /*
+ * Array of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+ #define ENCODED_STR_ARRAY_ALLOC_SIZE 128
+ StringInfo *str_set = NULL;
+ int str_set_index;
+ int str_set_size;
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ /* build encoded string array */
+ if (str_set == NULL)
+ {
+ str_set_index = 0;
+ str_set_size = ENCODED_STR_ARRAY_ALLOC_SIZE * sizeof(StringInfo);
+ str_set = palloc(str_set_size);
+ }
+
+ str_set[str_set_index++] = encoded_str;
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ if (str_set_index >= str_set_size)
+ {
+ str_set_size *= 2;
+ str_set = repalloc(str_set, str_set_size);
+ }
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (str_set == NULL)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set, str_set_index);
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search set of encode_str.
+ * set_size: size of set_str array.
+ */
+static
+int search_str_set(char *pattern, StringInfo *str_set, int set_size)
+{
+ char *encoded_str = palloc0(set_size+1);
+ int resultlen = 0;
+
+ search_str_set_recurse(pattern, str_set, set_size, 0, encoded_str, &resultlen);
+ elog(DEBUG1, "search_str_set returns %d", resultlen);
+ return resultlen;
+}
+
+/*
+ * Workhorse of search_str_set.
+ */
+static
+void search_str_set_recurse(char *pattern, StringInfo *str_set,
+ int set_size, int set_index, char *encoded_str, int *resultlen)
+{
+ char *p;
+
+ if (set_index >= set_size)
+ {
+ Datum d;
+ text *res;
+ char *substr;
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how log the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr is, it error
+ * out if pattern is not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ int len;
+
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ if (len > *resultlen)
+ /* remember the longest match */
+ *resultlen = len;
+ }
+ }
+ return;
+ }
+
+ p = str_set[set_index]->data;
+ while (*p)
+ {
+ encoded_str[set_index] = *p;
+ p++;
+ search_str_set_recurse(pattern, str_set, set_size, set_index + 1, encoded_str, resultlen);
+ }
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..d20f803cf5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..63feb68f60 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2471,6 +2471,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2519,6 +2524,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2569,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Tue_Sep_12_15_18_43_2023_359)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v6-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
@ 2023-09-15 10:02 Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir Belyalov @ 2023-09-15 10:02 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
> Since v5 patch failed applying anymore, updated the patch.
Thank you for updating the patch . I made a little review on it where
corrected some formatting.
> - COPY with a datatype error that can't be handled as a soft error
>
> I didn't know proper way to test this, but I've found data type widget's
> input function widget_in() defined to occur hard-error in regress.c,
> attached patch added a test using it.
>
This test seems to be weird a bit, because of the "widget" type. The hard
error is thrown by the previous test with missing data. Also it'll be
interesting for me to list all cases when a hard error can be thrown.
Regards,
Damir Belyalov
Postgres Professional
Attachments:
[text/x-patch] v7-0001-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch (12.1K, ../../CALH1Lgtg39crESw644KDH+ejBWuW3Vospe67F-ZWXNeBMEE=PA@mail.gmail.com/3-v7-0001-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch)
download | inline diff:
From 0e1193e00bb5ee810a015a2baaf7c79e395a54c7 Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Sep 2023 11:14:57 +0300
Subject: [PATCH] ignore errors
---
doc/src/sgml/ref/copy.sgml | 13 +++++++++
src/backend/commands/copy.c | 13 +++++++++
src/backend/commands/copyfrom.c | 37 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 20 ++++++++++---
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 3 ++
src/test/regress/expected/copy2.out | 28 ++++++++++++++++++
src/test/regress/sql/copy2.sql | 26 +++++++++++++++++
9 files changed, 139 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 4d614a0225..d5cdbb4025 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL ( <replaceable class="parameter">column_name</replaceable> [, ...] )
FORCE_NULL ( <replaceable class="parameter">column_name</replaceable> [, ...] )
+ IGNORE_DATATYPE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -370,6 +371,18 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_DATATYPE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ with columns where the data type's input-function raises an error.
+ This option is not allowed when using binary format. Note that this
+ is only supported in current <command>COPY</command> syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f14fae3308..beb73f5357 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool ignore_datatype_errors_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_datatype_errors") == 0)
+ {
+ if (ignore_datatype_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_datatype_errors_specified = true;
+ opts_out->ignore_datatype_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -594,6 +602,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->ignore_datatype_errors)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify IGNORE_DATATYPE_ERRORS in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 70871ed819..b18aea6376 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -752,6 +752,14 @@ CopyFrom(CopyFromState cstate)
ti_options |= TABLE_INSERT_FROZEN;
}
+ /* Set up soft error handler for IGNORE_DATATYPE_ERRORS */
+ if (cstate->opts.ignore_datatype_errors)
+ {
+ ErrorSaveContext escontext = {T_ErrorSaveContext};
+ escontext.details_wanted = true;
+ cstate->escontext = escontext;
+ }
+
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
@@ -987,7 +995,36 @@ CopyFrom(CopyFromState cstate)
/* Directly store the values/nulls array in the slot */
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ {
+ if (cstate->opts.ignore_datatype_errors &&
+ cstate->ignored_errors_count > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->ignored_errors_count));
break;
+ }
+
+ /* Soft error occured, skip this tuple and log the reason */
+ if (cstate->escontext.error_occurred)
+ {
+ ErrorSaveContext new_escontext = {T_ErrorSaveContext};
+
+ /* Adjust elevel so we don't jump out */
+ cstate->escontext.error_data->elevel = WARNING;
+
+ /*
+ * Despite the name, this won't raise an error since elevel is
+ * WARNING now.
+ */
+ ThrowErrorData(cstate->escontext.error_data);
+
+ ExecClearTuple(myslot);
+
+ new_escontext.details_wanted = true;
+ cstate->escontext = new_escontext;
+
+ continue;
+ }
ExecStoreVirtualTuple(myslot);
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f553734582..cf4dad1106 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -956,10 +957,21 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /*
+ * If IGNORE_DATATYPE_ERRORS is enabled, skip rows with
+ * datatype errors.
+ */
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) &cstate->escontext,
+ &values[m]))
+ {
+ cstate->ignored_errors_count++;
+
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 779fdc90cb..2fba51f648 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2869,7 +2869,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "IGNORE_DATATYPE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 33175868f6..c2e55ac21f 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_datatype_errors; /* ignore rows with datatype errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index ac2c16f8b8..e5bdae2d25 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,8 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext escontext; /* soft error trapper during in_functions execution */
+ int64 ignored_errors_count; /* total number of ignored errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index faf1a4d1b0..ac9c99f083 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -82,6 +82,8 @@ COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x to stdin (format BINARY, ignore_datatype_errors);
+ERROR: cannot specify IGNORE_DATATYPE_ERRORS in BINARY mode
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY force quote available only in CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -666,6 +668,30 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_DATATYPE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+WARNING: invalid input syntax for type integer: "a"
+WARNING: value "3333333333" is out of range for type integer
+WARNING: invalid input syntax for type integer: "a"
+WARNING: invalid input syntax for type integer: ""
+WARNING: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -680,6 +706,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index d759635068..e8c2c1aca3 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -70,6 +70,7 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x to stdin (format BINARY, ignore_datatype_errors);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
@@ -464,6 +465,29 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_DATATYPE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1 {1}
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -478,6 +502,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2023-09-19 14:00 ` torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2023-09-19 14:00 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
On 2023-09-15 19:02, Damir Belyalov wrote:
>> Since v5 patch failed applying anymore, updated the patch.
>
> Thank you for updating the patch . I made a little review on it where
> corrected some formatting.
>
Thanks for your review and update!
I don't have objections the modification of the codes and comments.
Although v7 patch doesn't have commit messages on the patch, I think
leave commit message is good for reviewers.
>>> - COPY with a datatype error that can't be handled as a soft error
>>
>> I didn't know proper way to test this, but I've found data type
>> widget's
>> input function widget_in() defined to occur hard-error in regress.c,
>> attached patch added a test using it.
>
> This test seems to be weird a bit, because of the "widget" type. The
> hard error is thrown by the previous test with missing data. Also
> it'll be interesting for me to list all cases when a hard error can be
> thrown.
Although missing data error is hard error, the suggestion from Andres
was adding `dataype` error:
> - COPY with a datatype error that can't be handled as a soft error
As described in widghet_in(), widget is intentionally left emitting hard
error for testing purpose:
> * Note: DON'T convert this error to "soft" style (errsave/ereturn).
> We
> * want this data type to stay permanently in the hard-error world so
> that
> * it can be used for testing that such cases still work reasonably.
From this point of view, I think this is a supposed way of using widget.
OTOH widget is declared in create_type.sql and I'm not sure it's ok to
use it in another test copy2.sql.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2023-09-20 16:15 ` Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Damir @ 2023-09-20 16:15 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
> Although v7 patch doesn't have commit messages on the patch, I think
> leave commit message is good for reviewers.
Sure, didn't notice it. Added the commit message to the updated patch.
> * Note: DON'T convert this error to "soft" style (errsave/ereturn). We
>> * want this data type to stay permanently in the hard-error world
>> so that
>> * it can be used for testing that such cases still work reasonably.
>
> From this point of view, I think this is a supposed way of using widget.
I agree, it's a good approach for checking datatype errors, because
that's what was intended.
> OTOH widget is declared in create_type.sql and I'm not sure it's ok to
> use it in another test copy2.sql.
I think that other regress tests with 'widget' type that will be created
in the future can be not only in the create_type.sql. So it's not a
problem that some type or function is taken from another regress test.
For example, the table 'onek' is used in many regress tests.
Regards,
Damir Belyalov
Postgres Professional
Attachments:
[text/x-patch] v7-0002-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch (12.4K, ../../[email protected]/2-v7-0002-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch)
download | inline diff:
From 0e1193e00bb5ee810a015a2baaf7c79e395a54c7 Mon Sep 17 00:00:00 2001
From: Damir Belyalov <[email protected]>
Date: Fri, 15 Sep 2023 11:14:57 +0300
Subject: [PATCH v7] Add new COPY option IGNORE_DATATYPE_ERRORS
Currently entire COPY fails even when there is one unexpected data
regarding data type or range.
IGNORE_DATATYPE_ERRORS ignores these errors and skips them and COPY
data which don't contain problem.
This patch uses the soft error handling infrastructure, which is
introduced by d9f7f5d32f20.
Author: Damir Belyalov, Atsushi Torikoshi
---
doc/src/sgml/ref/copy.sgml | 13 +++++++++
src/backend/commands/copy.c | 13 +++++++++
src/backend/commands/copyfrom.c | 37 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 20 ++++++++++---
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 3 ++
src/test/regress/expected/copy2.out | 28 ++++++++++++++++++
src/test/regress/sql/copy2.sql | 26 +++++++++++++++++
9 files changed, 139 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 4d614a0225..d5cdbb4025 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL ( <replaceable class="parameter">column_name</replaceable> [, ...] )
FORCE_NULL ( <replaceable class="parameter">column_name</replaceable> [, ...] )
+ IGNORE_DATATYPE_ERRORS [ <replaceable class="parameter">boolean</replaceable> ]
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -370,6 +371,18 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>IGNORE_DATATYPE_ERRORS</literal></term>
+ <listitem>
+ <para>
+ Drops rows that contain malformed data while copying. These are rows
+ with columns where the data type's input-function raises an error.
+ This option is not allowed when using binary format. Note that this
+ is only supported in current <command>COPY</command> syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index f14fae3308..beb73f5357 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool ignore_datatype_errors_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "ignore_datatype_errors") == 0)
+ {
+ if (ignore_datatype_errors_specified)
+ errorConflictingDefElem(defel, pstate);
+ ignore_datatype_errors_specified = true;
+ opts_out->ignore_datatype_errors = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -594,6 +602,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->ignore_datatype_errors)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify IGNORE_DATATYPE_ERRORS in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 70871ed819..b18aea6376 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -752,6 +752,14 @@ CopyFrom(CopyFromState cstate)
ti_options |= TABLE_INSERT_FROZEN;
}
+ /* Set up soft error handler for IGNORE_DATATYPE_ERRORS */
+ if (cstate->opts.ignore_datatype_errors)
+ {
+ ErrorSaveContext escontext = {T_ErrorSaveContext};
+ escontext.details_wanted = true;
+ cstate->escontext = escontext;
+ }
+
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
@@ -987,7 +995,36 @@ CopyFrom(CopyFromState cstate)
/* Directly store the values/nulls array in the slot */
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ {
+ if (cstate->opts.ignore_datatype_errors &&
+ cstate->ignored_errors_count > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->ignored_errors_count));
break;
+ }
+
+ /* Soft error occured, skip this tuple and log the reason */
+ if (cstate->escontext.error_occurred)
+ {
+ ErrorSaveContext new_escontext = {T_ErrorSaveContext};
+
+ /* Adjust elevel so we don't jump out */
+ cstate->escontext.error_data->elevel = WARNING;
+
+ /*
+ * Despite the name, this won't raise an error since elevel is
+ * WARNING now.
+ */
+ ThrowErrorData(cstate->escontext.error_data);
+
+ ExecClearTuple(myslot);
+
+ new_escontext.details_wanted = true;
+ cstate->escontext = new_escontext;
+
+ continue;
+ }
ExecStoreVirtualTuple(myslot);
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f553734582..cf4dad1106 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -956,10 +957,21 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /*
+ * If IGNORE_DATATYPE_ERRORS is enabled, skip rows with
+ * datatype errors.
+ */
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) &cstate->escontext,
+ &values[m]))
+ {
+ cstate->ignored_errors_count++;
+
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 779fdc90cb..2fba51f648 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2869,7 +2869,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "IGNORE_DATATYPE_ERRORS");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 33175868f6..c2e55ac21f 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -42,6 +42,7 @@ typedef struct CopyFormatOptions
* -1 if not specified */
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
+ bool ignore_datatype_errors; /* ignore rows with datatype errors */
bool csv_mode; /* Comma Separated Value format? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index ac2c16f8b8..e5bdae2d25 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,8 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext escontext; /* soft error trapper during in_functions execution */
+ int64 ignored_errors_count; /* total number of ignored errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index faf1a4d1b0..ac9c99f083 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -82,6 +82,8 @@ COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x to stdin (format BINARY, ignore_datatype_errors);
+ERROR: cannot specify IGNORE_DATATYPE_ERRORS in BINARY mode
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY force quote available only in CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -666,6 +668,30 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for IGNORE_DATATYPE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+WARNING: invalid input syntax for type integer: "a"
+WARNING: value "3333333333" is out of range for type integer
+WARNING: invalid input syntax for type integer: "a"
+WARNING: invalid input syntax for type integer: ""
+WARNING: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -680,6 +706,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index d759635068..e8c2c1aca3 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -70,6 +70,7 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x to stdin (format BINARY, ignore_datatype_errors);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
@@ -464,6 +465,29 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for IGNORE_DATATYPE_ERRORS option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (IGNORE_DATATYPE_ERRORS);
+1 {1}
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -478,6 +502,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
@ 2023-11-08 18:18 ` Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 23:53 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: Tom Lane @ 2023-11-08 18:18 UTC (permalink / raw)
To: Damir <[email protected]>; +Cc: torikoshia <[email protected]>; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Damir <[email protected]> writes:
> [ v7-0002-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch ]
Sorry for being so late to the party, but ... I don't think this
is a well-designed feature as it stands. Simply dropping failed rows
seems like an unusable definition for any application that has
pretensions of robustness. "But", you say, "we're emitting WARNING
messages about it". That's *useless*. For most applications WARNING
messages just go into the bit bucket, or worse they cause memory leaks
(because the app never reads them). An app that tried to read them
would have to cope with all sorts of fun such as translated messages.
Furthermore, as best I can tell from the provided test cases, the
messages completely lack basic context such as which field or line
the problem occurred in. An app trying to use this to understand
which input lines had failed would not get far.
I think an actually usable feature of this sort would involve
copying all the failed lines to some alternate output medium,
perhaps a second table with a TEXT column to receive the original
data line. (Or maybe an array of text that could receive the
broken-down field values?) Maybe we could dump the message info,
line number, field name etc into additional columns.
Also it'd be a good idea to have a vision of how the feature
could be extended to cope with lower-level errors, such as
lines that have the wrong number of columns or other problems
with line-level syntax. I don't say we need to cope with that
immediately, but it's going to be something people will want
to add, I think.
regards, tom lane
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2023-11-08 19:34 ` Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: Daniel Gustafsson @ 2023-11-08 19:34 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
> On 8 Nov 2023, at 19:18, Tom Lane <[email protected]> wrote:
> I think an actually usable feature of this sort would involve
> copying all the failed lines to some alternate output medium,
> perhaps a second table with a TEXT column to receive the original
> data line. (Or maybe an array of text that could receive the
> broken-down field values?) Maybe we could dump the message info,
> line number, field name etc into additional columns.
I agree that the errors should be easily visible to the user in some way. The
feature is for sure interesting, especially in data warehouse type jobs where
dirty data is often ingested.
As a data point, Greenplum has this feature with additional SQL syntax to
control it:
COPY .. LOG ERRORS SEGMENT REJECT LIMIT xyz ROWS;
LOG ERRORS instructs the database to log the faulty rows and SEGMENT REJECT
LIMIT xyz ROWS sets the limit of how many rows can be faulty before the
operation errors out. I'm not at all advocating that we should mimic this,
just wanted to add a reference to postgres derivative where this has been
implemented.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
@ 2023-11-08 20:12 ` Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-16 00:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: Tom Lane @ 2023-11-08 20:12 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Daniel Gustafsson <[email protected]> writes:
>> On 8 Nov 2023, at 19:18, Tom Lane <[email protected]> wrote:
>> I think an actually usable feature of this sort would involve
>> copying all the failed lines to some alternate output medium,
>> perhaps a second table with a TEXT column to receive the original
>> data line. (Or maybe an array of text that could receive the
>> broken-down field values?) Maybe we could dump the message info,
>> line number, field name etc into additional columns.
> I agree that the errors should be easily visible to the user in some way. The
> feature is for sure interesting, especially in data warehouse type jobs where
> dirty data is often ingested.
I agree it's interesting, but we need to get it right the first time.
Here is a very straw-man-level sketch of what I think might work.
The option to COPY FROM looks something like
ERRORS TO other_table_name (item [, item [, ...]])
where the "items" are keywords identifying the information item
we will insert into each successive column of the target table.
This design allows the user to decide which items are of use
to them. I envision items like
LINENO bigint COPY line number, counting from 1
LINE text raw text of line (after encoding conversion)
FIELDS text[] separated, de-escaped string fields (the data
that was or would be fed to input functions)
FIELD text name of troublesome field, if field-specific
MESSAGE text error message text
DETAIL text error message detail, if any
SQLSTATE text error SQLSTATE code
Some of these would have to be populated as NULL if we didn't get
that far in processing the line. In the worst case, which is
encoding conversion failure, I think we couldn't populate any of
the data items except LINENO.
Not sure if we need to insist that the target table columns be
exactly the data types I show above. It'd be nice to allow
the LINENO target to be plain int, perhaps. OTOH, do we really
want to have to deal with issues like conversion failures while
trying to report an error?
> As a data point, Greenplum has this feature with additional SQL syntax to
> control it:
> COPY .. LOG ERRORS SEGMENT REJECT LIMIT xyz ROWS;
> LOG ERRORS instructs the database to log the faulty rows and SEGMENT REJECT
> LIMIT xyz ROWS sets the limit of how many rows can be faulty before the
> operation errors out. I'm not at all advocating that we should mimic this,
> just wanted to add a reference to postgres derivative where this has been
> implemented.
Hm. A "reject limit" might be a useful add-on, but I wouldn't advocate
including it in the initial patch.
regards, tom lane
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2023-11-09 04:33 ` [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: [email protected] @ 2023-11-09 04:33 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; [email protected]
Tom Lane <[email protected]> writes:
> Daniel Gustafsson <[email protected]> writes:
>>> On 8 Nov 2023, at 19:18, Tom Lane <[email protected]> wrote:
>>> I think an actually usable feature of this sort would involve
>>> copying all the failed lines to some alternate output medium,
>>> perhaps a second table with a TEXT column to receive the original
>>> data line. (Or maybe an array of text that could receive the
>>> broken-down field values?) Maybe we could dump the message info,
>>> line number, field name etc into additional columns.
>
>> I agree that the errors should be easily visible to the user in some way. The
>> feature is for sure interesting, especially in data warehouse type jobs where
>> dirty data is often ingested.
>
> I agree it's interesting, but we need to get it right the first time.
>
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
> where the "items" are keywords identifying the information item
> we will insert into each successive column of the target table.
> This design allows the user to decide which items are of use
> to them. I envision items like
While I'm pretty happy with the overall design, which is 'ERRORS to
other_table_name' specially. I'm a bit confused why do we need to
write the codes for (item [, item [, ...]]), not only because it
requires more coding but also requires user to make more decisions.
will it be anything wrong to make all of them as default?
> LINENO bigint COPY line number, counting from 1
> LINE text raw text of line (after encoding conversion)
> FIELDS text[] separated, de-escaped string fields (the data
> that was or would be fed to input functions)
> FIELD text name of troublesome field, if field-specific
> MESSAGE text error message text
> DETAIL text error message detail, if any
> SQLSTATE text error SQLSTATE code
>
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
@ 2023-11-14 10:10 ` Damir Belyalov <[email protected]>
2023-11-14 10:16 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-11-15 01:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
0 siblings, 3 replies; 117+ messages in thread
From: Damir Belyalov @ 2023-11-14 10:10 UTC (permalink / raw)
To: [email protected]; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; [email protected]
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
I tried to implement the patch using a table and came across a number of
questions.
Which table should we implement for this feature: a system catalog table or
store this table as a file or create a new table?
In these cases, security and user rights management issues arise.
It is better for other users not to see error lines from another user. It
is also not clear how access rights to this table are inherited and be
given.
--
Regards,
Damir Belyalov
Postgres Professional
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2023-11-14 10:16 ` Alena Rybakina <[email protected]>
2 siblings, 0 replies; 117+ messages in thread
From: Alena Rybakina @ 2023-11-14 10:16 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; [email protected]; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; [email protected]
Hi!
On 14.11.2023 13:10, Damir Belyalov wrote:
>
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
>
> I tried to implement the patch using a table and came across a number
> of questions.
>
> Which table should we implement for this feature: a system catalog
> table or store this table as a file or create a new table?
>
> In these cases, security and user rights management issues arise.
> It is better for other users not to see error lines from another user.
> It is also not clear how access rights to this table are inherited and
> be given.
>
>
Maybe we can add a guc or a parameter to output such errors during the
execution of the copy function with errors and check whether the user
has enough rights to set such a parameter?
That is, I propose to give the user a choice to run copy with and
without saving errors and at the same time immediately check whether the
option with error output is possible for him in principle?
--
Regards,
Alena Rybakina
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2023-11-15 01:23 ` [email protected]
2 siblings, 0 replies; 117+ messages in thread
From: [email protected] @ 2023-11-15 01:23 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>; [email protected]
Damir Belyalov <[email protected]> writes:
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
> I tried to implement the patch using a table and came across a number of questions.
>
> Which table should we implement for this feature: a system catalog table or store this table as a file or create a new
> table?
I think system catalog should not be a option at the first place since
it requires more extra workload to do. see the calls of
IsCatalogRelation in heapam.c.
I prefer to create a new normal heap relation rather than a file since
heap realtion probabaly have better APIs.
> In these cases, security and user rights management issues arise.
> It is better for other users not to see error lines from another
> user. It is also not clear how access rights to this
> table are inherited and be given.
How about creating the table just allowing the current user to
read/write or just same as the relation we are copying to?
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
@ 2023-11-24 03:52 ` Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2 siblings, 1 reply; 117+ messages in thread
From: Andrei Lepikhov @ 2023-11-24 03:52 UTC (permalink / raw)
To: Damir Belyalov <[email protected]>; [email protected]; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Alena Rybakina <[email protected]>; [email protected]
On 14/11/2023 17:10, Damir Belyalov wrote:
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
>
> I tried to implement the patch using a table and came across a number of
> questions.
>
> Which table should we implement for this feature: a system catalog table
> or store this table as a file or create a new table?
>
> In these cases, security and user rights management issues arise.
> It is better for other users not to see error lines from another user.
> It is also not clear how access rights to this table are inherited and
> be given.
Previous reviews have given helpful ideas about storing errors in the
new table.
It should be trivial code - use the current table name + 'err' + suffix
as we already do in the case of conflicting auto-generated index names.
The 'errors table' must inherit any right policies from the table, to
which we do the copy.
--
regards,
Andrei Lepikhov
Postgres Professional
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
@ 2023-12-04 02:23 ` jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-04 02:23 UTC (permalink / raw)
To: Andrei Lepikhov <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Alena Rybakina <[email protected]>; [email protected]
hi.
here is my implementation based on previous discussions
add a new COPY FROM flag save_error.
save_error only works with non-BINARY flags.
save_error is easier for me to implement, if using "save error" I
worry, 2 words, gram.y will not work.
save_error also works other flag like {csv mode, force_null, force_not_null}
overall logic is:
if save_error is specified then
if error_holding table not exists then create one
if error_holding table exists set error_firsttime to false.
if save_error is not specified then work as master branch.
if errors happen then insert error info to error_holding table.
if errors do not exist and error_firsttime is true then drop the table.
if errors do not exist and error_firsttime is false then raise a
notice: All the past error holding saved at %s.%s
error holding table:
schema will be the same as COPY destination table.
the table name will be: COPY destination name concatenate with "_error".
error_holding table definition:
CREATE TABLE err_nsp.error_rel (LINENO BIGINT, LINE TEXT,
FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT,
ERR_DETAIL TEXT, ERRORCODE TEXT);
the following field is not implemented.
FIELDS text[], separated, de-escaped string fields (the data that was
or would be fed to input functions)
because imagine following case:
create type test as (a int, b text);
create table copy_comp (c1 int, c2 test default '(11,test)', c3 date);
copy copy_comp from stdin with (default '\D');
1 \D '2022-07-04'
\.
table copy_comp;
I feel it's hard from textual '\D' to get text[] `(11,test)` via SPI.
--------------------------------------
demo:
create table copy_default_error_save (
id integer,
text_value text not null default 'test',
ts_value timestamp without time zone not null default '2022-07-05'
);
copy copy_default_error_save from stdin with (save_error, default '\D');
k value '2022-07-04'
z \D '2022-07-03ASKL'
s \D \D
\.
NOTICE: 3 rows were skipped because of error. skipped row saved to
table public.copy_default_error_save_error
select * from copy_default_error_save_error;
lineno | line | field | source
| err_message |
err_detail | errorcode
--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
1 | k value '2022-07-04' | id | k
| invalid input syntax for type integer: "k" |
| 22P02
2 | z \D '2022-07-03ASKL' | id | z
| invalid input syntax for type integer: "z" |
| 22P02
2 | z \D '2022-07-03ASKL' | ts_value |
'2022-07-03ASKL' | invalid input syntax for type timestamp:
"'2022-07-03ASKL'" | | 22007
3 | s \D \D | id | s
| invalid input syntax for type integer: "s" |
| 22P02
(4 rows)
The doc is not so good.
COPY FROM (save_error), it will not be as fast as COPY FROM (save_error false).
With save_error, we can only use InputFunctionCallSafe, which I
believe is not as fast as InputFunctionCall.
If any conversion error happens, we need to call the SPI interface,
that would add more overhead. also we can only insert error cases row
by row. (maybe we can insert to error_save values(error1), (error2).
(I will try later)...
The main code is about constructing SPI query, and test and test output.
Attachments:
[text/x-patch] v8-0001-Add-a-new-COPY-option-SAVE_ERROR.patch (37.2K, ../../CACJufxGBb-WE7R1fM4kJxvmsgsyKL_4vce3s_KxLbo_ZEnFnWw@mail.gmail.com/2-v8-0001-Add-a-new-COPY-option-SAVE_ERROR.patch)
download | inline diff:
From 7aeb55cb0c8b1b36fd5c468fee0b07d4c13d1a7d Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Sun, 3 Dec 2023 22:58:40 +0800
Subject: [PATCH v8 1/1] Add a new COPY option: SAVE_ERROR. Only works for COPY
FROM, non-BINARY mode.
Currently NextCopyFrom can have 3 errors reported.
* extra data after last expected column
* missing data for column \"%s\"
* main function InputFunctionCall inside error.
Currently, we only deal with InputFunctionCall errors only.
instead of throw error while copying, save_error will save errors to a table automatically.
We check the table definition via column name and column data type.
if table already exists and meets the condition then errors will save to that table.
While copying, if error never happened, error save table will be dropped at the ending of COPY.
If the error saving table already exists,
meaning at least once COPY FROM errors had happened,
then all the future error will save to that table.
---
contrib/file_fdw/file_fdw.c | 4 +-
doc/src/sgml/ref/copy.sgml | 88 +++++++++++++
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 151 ++++++++++++++++++++++-
src/backend/commands/copyfromparse.c | 89 ++++++++++++-
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 3 +-
src/include/commands/copyfrom_internal.h | 7 ++
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 132 ++++++++++++++++++++
src/test/regress/sql/copy2.sql | 99 +++++++++++++++
12 files changed, 585 insertions(+), 12 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 2189be8a..2d3eb34f 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -751,7 +751,7 @@ fileIterateForeignScan(ForeignScanState *node)
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
found = NextCopyFrom(festate->cstate, econtext,
- slot->tts_values, slot->tts_isnull);
+ slot->tts_values, slot->tts_isnull, NULL);
if (found)
ExecStoreVirtualTuple(slot);
@@ -1183,7 +1183,7 @@ file_acquire_sample_rows(Relation onerel, int elevel,
MemoryContextReset(tupcontext);
MemoryContextSwitchTo(tupcontext);
- found = NextCopyFrom(cstate, NULL, values, nulls);
+ found = NextCopyFrom(cstate, NULL, values, nulls, NULL);
MemoryContextSwitchTo(oldcontext);
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..06096fa6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,17 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies malformed data make data type conversion failure while copying will automatically report error information to a regualar table.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -572,6 +584,13 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ if <literal>SAVE_ERROR</literal> spceicfied, error actually happened then
+ <productname>PostgreSQL</productname> will create one table for you, if no error happened
+ error_table not exist, nothing will happed.
+
+ </para>
+
</refsect1>
<refsect1>
@@ -962,6 +981,75 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title>Error Save Table </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> spceicfied, all the data type conversion fail while copying will automatically saved in a regular table.
+ <xref linkend="copy-errorsave-table"/> shows the error save table name, data type, and description.
+ </para>
+
+ <table id="copy-errorsave-table">
+
+ <title>COPY ERROR SAVE TABLE </title>
+
+ <tgroup cols="2">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry> Raw content of error occuring line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>field</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry> Field name of the error occuring </entry>
+ </row>
+
+ <row>
+ <entry> <literal>source</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry> Raw content of the error occuring field </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry> Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry> The error code for the copying error <literal>*</literal> </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..acd5b623 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -38,6 +38,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -652,10 +653,12 @@ CopyFrom(CopyFromState cstate)
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ StringInfo err_save_buf;
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -952,6 +955,7 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ err_save_buf = makeStringInfo();
for (;;)
{
TupleTableSlot *myslot;
@@ -989,8 +993,54 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
/* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull, err_save_buf))
+ {
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->error_nsp && cstate->error_rel);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%ld rows were skipped because of error."
+ " skipped row saved to table %s.%s",
+ cstate->error_rows_cnt,
+ cstate->error_nsp, cstate->error_rel));
+ }
+ else
+ {
+ StringInfoData querybuf;
+ if (cstate->error_firsttime)
+ {
+ ereport(NOTICE,
+ errmsg("No error happened."
+ "Error holding table %s.%s will be droped",
+ cstate->error_nsp, cstate->error_rel));
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DROP TABLE IF EXISTS %s.%s CASCADE ",
+ cstate->error_nsp, cstate->error_rel);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+ }
+ else
+ ereport(NOTICE,
+ errmsg("No error happened. "
+ "All the past error holding saved at %s.%s ",
+ cstate->error_nsp, cstate->error_rel));
+ }
+ }
break;
+ }
+
+ /* Soft error occured, skip this tuple */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
ExecStoreVirtualTuple(myslot);
@@ -1444,6 +1494,103 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ char *err_nsp;
+ char error_rel[NAMEDATALEN];
+ StringInfoData querybuf;
+ bool isnull;
+ bool error_table_ok;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ snprintf(error_rel, sizeof(error_rel), "%s",
+ RelationGetRelationName(cstate->rel));
+ strlcat(error_rel,"_error", NAMEDATALEN);
+ err_nsp = get_namespace_name(RelationGetNamespace(cstate->rel));
+
+ initStringInfo(&querybuf);
+ /* The build query is used to validate:
+ * . err_nsp.error_rel table exists
+ * . column list(order by attnum, begin from ctid) =
+ * {ctid, lineno,line,field,source,err_message,err_detail,errorcode}
+ * . data types (from attnum = -1) ={tid, int8,text,text,text,text,text,text}
+ * We need ctid system column when
+ * save_error table already exists and have zero column.
+ *
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,lineno,line,field,source,err_message,err_detail,errorcode}') AND "
+ "(array_agg(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+
+ appendStringInfo(&querybuf,
+ "relname = $$%s$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ error_rel, err_nsp);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ error_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ /* no err_nsp.error_rel table then crete one. for holding error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.%s (LINENO BIGINT, LINE TEXT, "
+ "FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT, "
+ "ERR_DETAIL TEXT, ERRORCODE TEXT)",
+ err_nsp,error_rel);
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ cstate->error_firsttime = true;
+ elog(DEBUG1, "%s.%s created ", err_nsp, error_rel);
+ }
+ else if (error_table_ok)
+ /* error save table already exists. Set error_firsttime to false */
+ cstate->error_firsttime = false;
+ else if(!error_table_ok)
+ ereport(ERROR,
+ (errmsg("Error save table %s.%s already exists. "
+ "Cannot use it for COPY FROM error saving",
+ err_nsp, error_rel)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* these info need, no error will drop err_nsp.error_rel table */
+ cstate->error_rel = pstrdup(error_rel);
+ cstate->error_nsp = err_nsp;
+ }
+ else
+ {
+ /* set to NULL */
+ cstate->error_rel = NULL;
+ cstate->error_nsp = NULL;
+ cstate->escontext = NULL;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..5b5471af 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -66,10 +66,12 @@
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -852,7 +854,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
*/
bool
NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls)
+ Datum *values, bool *nulls, StringInfo err_save_buf)
{
TupleDesc tupDesc;
AttrNumber num_phys_attrs,
@@ -885,6 +887,11 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ /* reset to false for next new line if SAVE_ERROR specified */
+ if (cstate->opts.save_error)
+ {
+ cstate->line_error_occured = false;
+ }
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
ereport(ERROR,
@@ -956,15 +963,87 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ * So there is two function.
+ */
+ if(!cstate->opts.save_error)
+ {
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ }
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char errcode[12];
+ char *err_detail;
+ snprintf(errcode, sizeof(errcode),
+ "%s",
+ unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ resetStringInfo(err_save_buf);
+ /* error table first column is bigint, reset is text.*/
+ appendStringInfo(err_save_buf,
+ "insert into %s.%s(lineno,line,field, "
+ "source, err_message, errorcode,err_detail) "
+ "select $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$, $$%s$$, ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->cur_lineno, cstate->line_buf.data,
+ cstate->cur_attname, string,
+ cstate->escontext->error_data->message,
+ errcode);
+
+ if (!err_detail)
+ appendStringInfo(err_save_buf, "NULL::text");
+ else
+ appendStringInfo(err_save_buf,"$$%s$$", err_detail);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+ if (SPI_processed != 1)
+ elog(FATAL, "not a singleton result");
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d631ac89..747bd88a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVE_ERROR SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17328,6 +17332,7 @@ unreserved_keyword:
| ROUTINES
| ROWS
| RULE
+ | SAVE_ERROR
| SAVEPOINT
| SCALAR
| SCHEMA
@@ -17936,6 +17941,7 @@ bare_label_keyword:
| ROW
| ROWS
| RULE
+ | SAVE_ERROR
| SAVEPOINT
| SCALAR
| SCHEMA
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..cfed5d7f 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to another table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
@@ -82,7 +83,7 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls);
+ Datum *values, bool *nulls, StringInfo err_save_buf);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..b1c02b2f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,12 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ int64 error_rows_cnt; /* total number of rows that have errors */
+ const char *error_rel; /* the error row save table name */
+ const char *error_nsp; /* the error row table's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
+ bool error_firsttime; /* first time create error save table */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..0906cc40 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,113 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+NOTICE: No error happened.Error holding table public.save_error_csv_error will be droped
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+ expected_zero
+---------------
+ 0
+(1 row)
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+create table save_error_csv_error();
+--should fail. since error save table already exists.
+--error save table name = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: Error save table public.save_error_csv_error already exists. Cannot use it for COPY FROM error saving
+DROP TABLE save_error_csv_error;
+BEGIN;
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+SELECT *, b is null as b_null, b = '' as empty FROM save_error_csv;
+ a | b | c | d | b_null | empty
+---+---+------+------+--------+-------
+ 2 | | NULL | NULL | f | t
+(1 row)
+
+SELECT count(*) as expect_one FROM pg_class WHERE relname = 'save_error_csv_error';
+ expect_one
+------------
+ 1
+(1 row)
+
+ROLLBACK;
+DROP TABLE save_error_csv;
+--error TABLE should already droppped.
+SELECT 1 as expect_zero FROM pg_class WHERE relname = 'save_error_csv_error';
+ expect_zero
+-------------
+(0 rows)
+
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 8 rows were skipped because of error. skipped row saved to table public.check_ign_err_error
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+NOTICE: No error happened. All the past error holding saved at public.check_ign_err_error
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+ lineno | line | field | source | err_message | err_detail | errorcode
+--------+--------------------------------------------+-------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ 2 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | " | |
+ 3 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ 4 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ 5 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ 6 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ 6 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ 7 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ 7 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ 7 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ 8 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ 8 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ 9 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ 9 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ 9 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(14 rows)
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY check_ign_err FROM STDIN WITH (save_error, save_error o...
+ ^
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 4 rows were skipped because of error. skipped row saved to table public.textrange_input_error
+SELECT * FROM textrange_input_error;
+ lineno | line | field | source | err_message | err_detail | errorcode
+--------+----------------------------+-------+----------+-------------------------------------------------------------------+------------------------------------------+-----------
+ 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ 2 | (",a),(",",a),()",a) | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ 2 | (",a),(",",a),()",a) | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ 2 | (",a),(",",a),()",a) | c | a) | malformed range literal: "a)" | Missing left parenthesis or bracket. | 22P02
+ 3 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ 3 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ 3 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ 4 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ 4 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ 4 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +929,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of error. skipped row saved to table public.copy_default_error_save_error
+select count(*) as expect_zero from copy_default_error_save;
+ expect_zero
+-------------
+ 0
+(1 row)
+
+select * from copy_default_error_save_error;
+ lineno | line | field | source | err_message | err_detail | errorcode
+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..3f8137cf 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,89 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+\.
+
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+create table save_error_csv_error();
+--should fail. since error save table already exists.
+--error save table name = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+DROP TABLE save_error_csv_error;
+
+BEGIN;
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as empty FROM save_error_csv;
+SELECT count(*) as expect_one FROM pg_class WHERE relname = 'save_error_csv_error';
+ROLLBACK;
+
+DROP TABLE save_error_csv;
+
+--error TABLE should already droppped.
+SELECT 1 as expect_zero FROM pg_class WHERE relname = 'save_error_csv_error';
+
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+,,,
+\.
+
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a)
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT * FROM textrange_input_error;
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
+
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +692,19 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+select count(*) as expect_zero from copy_default_error_save;
+select * from copy_default_error_save_error;
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-05 10:07 ` Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alena Rybakina @ 2023-12-05 10:07 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi!
Thank you for your contribution to this thread.
On 04.12.2023 05:23, jian he wrote:
> hi.
> here is my implementation based on previous discussions
>
> add a new COPY FROM flag save_error.
> save_error only works with non-BINARY flags.
> save_error is easier for me to implement, if using "save error" I
> worry, 2 words, gram.y will not work.
> save_error also works other flag like {csv mode, force_null, force_not_null}
>
> overall logic is:
> if save_error is specified then
> if error_holding table not exists then create one
> if error_holding table exists set error_firsttime to false.
> if save_error is not specified then work as master branch.
>
> if errors happen then insert error info to error_holding table.
> if errors do not exist and error_firsttime is true then drop the table.
> if errors do not exist and error_firsttime is false then raise a
> notice: All the past error holding saved at %s.%s
>
> error holding table:
> schema will be the same as COPY destination table.
> the table name will be: COPY destination name concatenate with "_error".
>
> error_holding table definition:
> CREATE TABLE err_nsp.error_rel (LINENO BIGINT, LINE TEXT,
> FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT,
> ERR_DETAIL TEXT, ERRORCODE TEXT);
>
> the following field is not implemented.
> FIELDS text[], separated, de-escaped string fields (the data that was
> or would be fed to input functions)
>
> because imagine following case:
> create type test as (a int, b text);
> create table copy_comp (c1 int, c2 test default '(11,test)', c3 date);
> copy copy_comp from stdin with (default '\D');
> 1 \D '2022-07-04'
> \.
> table copy_comp;
>
> I feel it's hard from textual '\D' to get text[] `(11,test)` via SPI.
> --------------------------------------
> demo:
>
> create table copy_default_error_save (
> id integer,
> text_value text not null default 'test',
> ts_value timestamp without time zone not null default '2022-07-05'
> );
> copy copy_default_error_save from stdin with (save_error, default '\D');
> k value '2022-07-04'
> z \D '2022-07-03ASKL'
> s \D \D
> \.
>
> NOTICE: 3 rows were skipped because of error. skipped row saved to
> table public.copy_default_error_save_error
> select * from copy_default_error_save_error;
> lineno | line | field | source
> | err_message |
> err_detail | errorcode
> --------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
> 1 | k value '2022-07-04' | id | k
> | invalid input syntax for type integer: "k" |
> | 22P02
> 2 | z \D '2022-07-03ASKL' | id | z
> | invalid input syntax for type integer: "z" |
> | 22P02
> 2 | z \D '2022-07-03ASKL' | ts_value |
> '2022-07-03ASKL' | invalid input syntax for type timestamp:
> "'2022-07-03ASKL'" | | 22007
> 3 | s \D \D | id | s
> | invalid input syntax for type integer: "s" |
> | 22P02
> (4 rows)
>
> The doc is not so good.
>
> COPY FROM (save_error), it will not be as fast as COPY FROM (save_error false).
> With save_error, we can only use InputFunctionCallSafe, which I
> believe is not as fast as InputFunctionCall.
> If any conversion error happens, we need to call the SPI interface,
> that would add more overhead. also we can only insert error cases row
> by row. (maybe we can insert to error_save values(error1), (error2).
> (I will try later)...
>
> The main code is about constructing SPI query, and test and test output.
I reviewed it and have a few questions.
1. I have seen that you delete a table before creating it, to which you
want to add errors due to a failed "copy from" operation. I think this
is wrong because this table can save useful data for the user.
At a minimum, we should warn the user about this, but I think we can
just add some number at the end of the name, such as name_table1,
name_table_2.
2. I noticed that you are forming a table name using the type of errors
that prevent rows from being added during 'copy from' operation.
I think it would be better to use the name of the source file that was
used while 'copy from' was running.
In addition, there may be several such files, it is also worth considering.
3. I found spelling:
/* no err_nsp.error_rel table then crete one. for holding error. */
4. Maybe rewrite this comment
these info need, no error will drop err_nsp.error_rel table
to:
this information is necessary, no error will lead to the deletion of the
err_sp.error_rel table.
5. Is this part of the comment needed? I think it duplicates the
information below when we form the query.
* . column list(order by attnum, begin from ctid) =
* {ctid, lineno,line,field,source,err_message,err_detail,errorcode}
* . data types (from attnum = -1) ={tid,
int8,text,text,text,text,text,text}
I'm not sure if we need to order the rows by number. It might be easier
to work with these lines in the order they appear.
--
Regards,
Alena Rybakina
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
@ 2023-12-06 10:47 ` jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-06 10:47 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Tue, Dec 5, 2023 at 6:07 PM Alena Rybakina <[email protected]> wrote:
>
> Hi!
>
> Thank you for your contribution to this thread.
>
>
> I reviewed it and have a few questions.
>
> 1. I have seen that you delete a table before creating it, to which you want to add errors due to a failed "copy from" operation. I think this is wrong because this table can save useful data for the user.
> At a minimum, we should warn the user about this, but I think we can just add some number at the end of the name, such as name_table1, name_table_2.
Sorry. I don't understand this part.
Currently, if the error table name already exists, then the copy will
fail, an error will be reported.
I try to first create a table, if no error then the error table will be dropped.
Can you demo the expected behavior?
> 2. I noticed that you are forming a table name using the type of errors that prevent rows from being added during 'copy from' operation.
> I think it would be better to use the name of the source file that was used while 'copy from' was running.
> In addition, there may be several such files, it is also worth considering.
>
Another column added.
now it looks like:
SELECT * FROM save_error_csv_error;
filename | lineno | line
| field | source | err_message |
err_detail | errorcode
----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
STDIN | 1 | 2002 232 40 50 60 70
80 | NULL | NULL | extra data after last expected column |
NULL | 22P04
STDIN | 1 | 2000 230 23
| d | NULL | missing data for column "d" | NULL
| 22P04
STDIN | 1 | z,,""
| a | z | invalid input syntax for type integer: "z" | NULL
| 22P02
STDIN | 2 | \0,,
| a | \0 | invalid input syntax for type integer: "\0" | NULL
| 22P02
> 3. I found spelling:
>
> /* no err_nsp.error_rel table then crete one. for holding error. */
>
fixed.
> 4. Maybe rewrite this comment
>
> these info need, no error will drop err_nsp.error_rel table
> to:
> this information is necessary, no error will lead to the deletion of the err_sp.error_rel table.
>
fixed.
> 5. Is this part of the comment needed? I think it duplicates the information below when we form the query.
>
> * . column list(order by attnum, begin from ctid) =
> * {ctid, lineno,line,field,source,err_message,err_detail,errorcode}
> * . data types (from attnum = -1) ={tid, int8,text,text,text,text,text,text}
>
> I'm not sure if we need to order the rows by number. It might be easier to work with these lines in the order they appear.
>
Simplified the comment. "order by attnum" is to make sure that if
there is a table already existing, and the column name is like X and
the data type like Y, then we consider this table is good for holding
potential error info.
COPY FROM, main entry point is NextCopyFrom.
Now for non-binary mode, if you specified save_error then it will not
fail at NextCopyFrom.
all these three errors will be tolerated: extra data after last
expected column, missing data for column, data type conversion.
Attachments:
[text/x-patch] v9-0001-Make-COPY-FROM-more-error-tolerant.patch (42.0K, ../../CACJufxH3KGOGhnS-J2bYnA-mpABg4HG9Xddfj9zBBgoK+iGSsQ@mail.gmail.com/2-v9-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 990e1e0f5130431cf32069963bb980bb0692ce0b Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Wed, 6 Dec 2023 18:26:32 +0800
Subject: [PATCH v9 1/1] Make COPY FROM more error tolerant
Currently COPY FROM has 3 types of error while processing the source file.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error will save errors to a table automatically.
We check the table definition via column name and column data type.
if table already exists and meets the criteria then errors will save to that table.
if the table does not exist, then create one.
Only works for COPY FROM, non-BINARY mode.
While copying, if error never happened, error save table will be dropped at the ending of COPY FROM.
If the error saving table already exists, meaning at least once COPY FROM errors has happened,
then all the future errors will be saved to that table.
we save the error to error saving table using SPI, construct a query, then execute the query.
---
contrib/file_fdw/file_fdw.c | 4 +-
doc/src/sgml/ref/copy.sgml | 93 ++++++++++++
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 147 ++++++++++++++++++-
src/backend/commands/copyfromparse.c | 171 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 3 +-
src/include/commands/copyfrom_internal.h | 7 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 135 ++++++++++++++++++
src/test/regress/sql/copy2.sql | 108 ++++++++++++++
12 files changed, 673 insertions(+), 19 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 2189be8a..2d3eb34f 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -751,7 +751,7 @@ fileIterateForeignScan(ForeignScanState *node)
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
found = NextCopyFrom(festate->cstate, econtext,
- slot->tts_values, slot->tts_isnull);
+ slot->tts_values, slot->tts_isnull, NULL);
if (found)
ExecStoreVirtualTuple(slot);
@@ -1183,7 +1183,7 @@ file_acquire_sample_rows(Relation onerel, int elevel,
MemoryContextReset(tupcontext);
MemoryContextSwitchTo(tupcontext);
- found = NextCopyFrom(cstate, NULL, values, nulls);
+ found = NextCopyFrom(cstate, NULL, values, nulls, NULL);
MemoryContextSwitchTo(oldcontext);
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..a6370c42 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,17 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion failure while copying will automatically report error information to a regular table.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -572,6 +584,12 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is spceified and a conversion error occurs while copying, then
+ <productname>PostgreSQL</productname> will create a table to save all the conversion errors. Conversion error
+ include data type conversion failure, extra data or missing data in the source file.
+ </para>
+
</refsect1>
<refsect1>
@@ -962,6 +980,81 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title>Error Save Table </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> spceicfied, all the data type conversion fail while copying will automatically saved in a regular table.
+ <xref linkend="copy-errorsave-table"/> shows the error save table name, data type, and description.
+ </para>
+
+ <table id="copy-errorsave-table">
+
+ <title>COPY ERROR SAVE TABLE </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the input file</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of error occuring line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>field</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field name of the error occuring</entry>
+ </row>
+
+ <row>
+ <entry> <literal>source</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occuring field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code for the copying error</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..ee6f2664 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -38,6 +38,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -652,10 +653,12 @@ CopyFrom(CopyFromState cstate)
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ StringInfo err_save_buf;
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -952,6 +955,7 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ err_save_buf = makeStringInfo();
for (;;)
{
TupleTableSlot *myslot;
@@ -989,8 +993,54 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
/* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull, err_save_buf))
+ {
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->error_nsp && cstate->error_rel);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%ld rows were skipped because of error."
+ " skipped row saved to table %s.%s",
+ cstate->error_rows_cnt,
+ cstate->error_nsp, cstate->error_rel));
+ }
+ else
+ {
+ StringInfoData querybuf;
+ if (cstate->error_firsttime)
+ {
+ ereport(NOTICE,
+ errmsg("No error happened."
+ "Error holding table %s.%s will be droped",
+ cstate->error_nsp, cstate->error_rel));
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DROP TABLE IF EXISTS %s.%s CASCADE ",
+ cstate->error_nsp, cstate->error_rel);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+ }
+ else
+ ereport(NOTICE,
+ errmsg("No error happened. "
+ "All the past error holding saved at %s.%s ",
+ cstate->error_nsp, cstate->error_rel));
+ }
+ }
break;
+ }
+
+ /* Soft error occured, skip this tuple */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
ExecStoreVirtualTuple(myslot);
@@ -1444,6 +1494,99 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ char *err_nsp;
+ char error_rel[NAMEDATALEN];
+ StringInfoData querybuf;
+ bool isnull;
+ bool error_table_ok;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ snprintf(error_rel, sizeof(error_rel), "%s",
+ RelationGetRelationName(cstate->rel));
+ strlcat(error_rel,"_error", NAMEDATALEN);
+ err_nsp = get_namespace_name(RelationGetNamespace(cstate->rel));
+
+ initStringInfo(&querybuf);
+ /* The build query is used to validate:
+ * Does err_nsp.error_rel table exist?
+ * if err_nsp.error_rel exists, does it meet our criteria?
+ * our criteria of error table is based on column name and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,filename,lineno,line,field,source,err_message,err_detail,errorcode}') AND "
+ "(array_agg(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+
+ appendStringInfo(&querybuf,
+ "relname = $$%s$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ error_rel, err_nsp);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ error_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ /* No err_nsp.error_rel table then create it for holding error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.%s (FILENAME TEXT, LINENO BIGINT, LINE TEXT, "
+ "FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT, "
+ "ERR_DETAIL TEXT, ERRORCODE TEXT)",
+ err_nsp,error_rel);
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ cstate->error_firsttime = true;
+ elog(DEBUG1, "%s.%s created ", err_nsp, error_rel);
+ }
+ else if (error_table_ok)
+ /* error save table already exists. Set error_firsttime to false */
+ cstate->error_firsttime = false;
+ else if(!error_table_ok)
+ ereport(ERROR,
+ (errmsg("Error save table %s.%s already exists. "
+ "Cannot use it for COPY FROM error saving",
+ err_nsp, error_rel)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* thses information is necessary, no error then drop err_sp.error_rel table*/
+ cstate->error_rel = pstrdup(error_rel);
+ cstate->error_nsp = err_nsp;
+ }
+ else
+ {
+ /* set to NULL */
+ cstate->error_rel = NULL;
+ cstate->error_nsp = NULL;
+ cstate->escontext = NULL;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..d7ddf64c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -66,10 +66,12 @@
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -852,7 +854,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
*/
bool
NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls)
+ Datum *values, bool *nulls, StringInfo err_save_buf)
{
TupleDesc tupDesc;
AttrNumber num_phys_attrs,
@@ -885,11 +887,48 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ /* reset to false for next new line if SAVE_ERROR specified */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ char *errmsg_extra = "extra data after last expected column";
+
+ resetStringInfo(err_save_buf);
+ /* add line buf, etc for line have extra data to error save table*/
+ appendStringInfo(err_save_buf,
+ "insert into %s.%s(filename, lineno,line, "
+ "err_message, errorcode) "
+ "select $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ errmsg_extra,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +940,46 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ char errmsg[128];
+ snprintf(errmsg, sizeof(errmsg),
+ "missing data for column \"%s\"",
+ NameStr(att->attname));
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "insert into %s.%s(filename,lineno,line, field, "
+ "err_message, errorcode) "
+ "select $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$ ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ NameStr(att->attname), errmsg,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1031,87 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ * So there is two function.
+ */
+ if(!cstate->opts.save_error)
+ {
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ }
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char errcode[12];
+ char *err_detail;
+ snprintf(errcode, sizeof(errcode),
+ "%s",
+ unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ resetStringInfo(err_save_buf);
+ /* error table first column is bigint, reset is text.*/
+ appendStringInfo(err_save_buf,
+ "insert into %s.%s(filename, lineno,line,field, "
+ "source, err_message, errorcode,err_detail) "
+ "select $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$, $$%s$$, ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ cstate->cur_attname, string,
+ cstate->escontext->error_data->message,
+ errcode);
+
+ if (!err_detail)
+ appendStringInfo(err_save_buf, "NULL::text");
+ else
+ appendStringInfo(err_save_buf,"$$%s$$", err_detail);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d631ac89..747bd88a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVE_ERROR SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17328,6 +17332,7 @@ unreserved_keyword:
| ROUTINES
| ROWS
| RULE
+ | SAVE_ERROR
| SAVEPOINT
| SCALAR
| SCHEMA
@@ -17936,6 +17941,7 @@ bare_label_keyword:
| ROW
| ROWS
| RULE
+ | SAVE_ERROR
| SAVEPOINT
| SCALAR
| SCHEMA
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..de47791a 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
@@ -82,7 +83,7 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls);
+ Datum *values, bool *nulls, StringInfo err_save_buf);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..b1c02b2f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,12 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ int64 error_rows_cnt; /* total number of rows that have errors */
+ const char *error_rel; /* the error row save table name */
+ const char *error_nsp; /* the error row table's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
+ bool error_firsttime; /* first time create error save table */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..bb86bb9f 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,116 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+NOTICE: No error happened.Error holding table public.save_error_csv_error will be droped
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+ expected_zero
+---------------
+ 0
+(1 row)
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error) already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: Error save table public.save_error_csv_error already exists. Cannot use it for COPY FROM error saving
+DROP TABLE save_error_csv_error;
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | d | b_null | b_empty
+---+---+------+------+--------+---------
+ 2 | | NULL | NULL | f | t
+(1 row)
+
+SELECT * FROM save_error_csv_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
+ STDIN | 1 | 2002 232 40 50 60 70 80 | NULL | NULL | extra data after last expected column | NULL | 22P04
+ STDIN | 1 | 2000 230 23 | d | NULL | missing data for column "d" | NULL | 22P04
+ STDIN | 1 | z,,"" | a | z | invalid input syntax for type integer: "z" | NULL | 22P02
+ STDIN | 2 | \0,, | a | \0 | invalid input syntax for type integer: "\0" | NULL | 22P02
+(4 rows)
+
+DROP TABLE save_error_csv, save_error_csv_error;
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 8 rows were skipped because of error. skipped row saved to table public.check_ign_err_error
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+NOTICE: No error happened. All the past error holding saved at public.check_ign_err_error
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+--------------------------------------------+-------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ STDIN | 2 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | " | |
+ STDIN | 3 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 4 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ STDIN | 5 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 9 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(14 rows)
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY check_ign_err FROM STDIN WITH (save_error, save_error o...
+ ^
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 4 rows were skipped because of error. skipped row saved to table public.textrange_input_error
+SELECT * FROM textrange_input_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------+-------+----------+-------------------------------------------------------------------+------------------------------------------+-----------
+ STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 2 | (",a),(",",a),()",a) | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | c | a) | malformed range literal: "a)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 4 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +932,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of error. skipped row saved to table public.copy_default_error_save_error
+select count(*) as expect_zero from copy_default_error_save;
+ expect_zero
+-------------
+ 0
+(1 row)
+
+select * from copy_default_error_save_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..8c8d8adb 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,98 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+\.
+
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error) already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+DROP TABLE save_error_csv_error;
+
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+2002 232 40 50 60 70 80
+\.
+
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+2000 230 23
+\.
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+
+SELECT * FROM save_error_csv_error;
+
+DROP TABLE save_error_csv, save_error_csv_error;
+
+
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+,,,
+\.
+
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a)
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT * FROM textrange_input_error;
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
+
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +701,19 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+select count(*) as expect_zero from copy_default_error_save;
+select * from copy_default_error_save_error;
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-08 07:09 ` Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alena Rybakina @ 2023-12-08 07:09 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Thank you for your work. Unfortunately, your code contained errors
during the make installation:
'SAVEPOINT' after 'SAVE_ERROR' in unreserved_keyword list is misplaced
'SAVEPOINT' after 'SAVE_ERROR' in bare_label_keyword list is misplaced
make[2]: *** [../../../src/Makefile.global:783: gram.c] Error 1
make[1]: *** [Makefile:131: parser/gram.h] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [src/Makefile.global:383: submake-generated-headers] Error 2
I have ubuntu 22.04 operation system.
On 06.12.2023 13:47, jian he wrote:
> On Tue, Dec 5, 2023 at 6:07 PM Alena Rybakina<[email protected]> wrote:
>> Hi!
>>
>> Thank you for your contribution to this thread.
>>
>>
>> I reviewed it and have a few questions.
>>
>> 1. I have seen that you delete a table before creating it, to which you want to add errors due to a failed "copy from" operation. I think this is wrong because this table can save useful data for the user.
>> At a minimum, we should warn the user about this, but I think we can just add some number at the end of the name, such as name_table1, name_table_2.
> Sorry. I don't understand this part.
> Currently, if the error table name already exists, then the copy will
> fail, an error will be reported.
> I try to first create a table, if no error then the error table will be dropped.
To be honest, first of all, I misunderstood this part of the code. Now I
see that it works the way you mentioned.
However, I didn't see if you dealt with cases where we already had a
table with the same name as the table error.
I mean, when is he trying to create for the first time, or will we never
be able to face such a problem?
> Can you demo the expected behavior?
Unfortunately, I was unable to launch it due to a build issue.
>
>> 2. I noticed that you are forming a table name using the type of errors that prevent rows from being added during 'copy from' operation.
>> I think it would be better to use the name of the source file that was used while 'copy from' was running.
>> In addition, there may be several such files, it is also worth considering.
>>
> Another column added.
> now it looks like:
>
> SELECT * FROM save_error_csv_error;
> filename | lineno | line
> | field | source | err_message |
> err_detail | errorcode
> ----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
> STDIN | 1 | 2002 232 40 50 60 70
> 80 | NULL | NULL | extra data after last expected column |
> NULL | 22P04
> STDIN | 1 | 2000 230 23
> | d | NULL | missing data for column "d" | NULL
> | 22P04
> STDIN | 1 | z,,""
> | a | z | invalid input syntax for type integer: "z" | NULL
> | 22P02
> STDIN | 2 | \0,,
> | a | \0 | invalid input syntax for type integer: "\0" | NULL
> | 22P02
>
Yes, I see the "filename" column, and this will solve the problem, but
"STDIN" is unclear to me.
>> 3. I found spelling:
>>
>> /* no err_nsp.error_rel table then crete one. for holding error. */
>>
> fixed.
>
>> 4. Maybe rewrite this comment
>>
>> these info need, no error will drop err_nsp.error_rel table
>> to:
>> this information is necessary, no error will lead to the deletion of the err_sp.error_rel table.
>>
> fixed.
Thank you.
>> 5. Is this part of the comment needed? I think it duplicates the information below when we form the query.
>>
>> * . column list(order by attnum, begin from ctid) =
>> * {ctid, lineno,line,field,source,err_message,err_detail,errorcode}
>> * . data types (from attnum = -1) ={tid, int8,text,text,text,text,text,text}
>>
>> I'm not sure if we need to order the rows by number. It might be easier to work with these lines in the order they appear.
>>
> Simplified the comment. "order by attnum" is to make sure that if
> there is a table already existing, and the column name is like X and
> the data type like Y, then we consider this table is good for holding
> potential error info.
>
> COPY FROM, main entry point is NextCopyFrom.
> Now for non-binary mode, if you specified save_error then it will not
> fail at NextCopyFrom.
> all these three errors will be tolerated: extra data after last
> expected column, missing data for column, data type conversion.
It looks clearer and better, thanks!
Comments in the format of questions are unusual for me, I perceive them
to think about it, for example, as here (contrib/bloom/blinsert.c:312):
/*
* Didn't find place to insert in notFullPage array. Allocate new page.
* (XXX is it good to do this while holding ex-lock on the metapage??)
*/
Maybe we can rewrite it like this:
/* Check, the err_nsp.error_rel table has already existed
* and if it is, check its column name and data types.
--
Regards,
Alena Rybakina
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
@ 2023-12-10 10:32 ` jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-10 10:32 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Fri, Dec 8, 2023 at 3:09 PM Alena Rybakina <[email protected]> wrote:
>
> Thank you for your work. Unfortunately, your code contained errors during the make installation:
>
> 'SAVEPOINT' after 'SAVE_ERROR' in unreserved_keyword list is misplaced
> 'SAVEPOINT' after 'SAVE_ERROR' in bare_label_keyword list is misplaced
> make[2]: *** [../../../src/Makefile.global:783: gram.c] Error 1
> make[1]: *** [Makefile:131: parser/gram.h] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make: *** [src/Makefile.global:383: submake-generated-headers] Error 2
>
> I have ubuntu 22.04 operation system.
>
> On 06.12.2023 13:47, jian he wrote:
>
> On Tue, Dec 5, 2023 at 6:07 PM Alena Rybakina <[email protected]> wrote:
>
> Hi!
>
> Thank you for your contribution to this thread.
>
>
> I reviewed it and have a few questions.
>
> 1. I have seen that you delete a table before creating it, to which you want to add errors due to a failed "copy from" operation. I think this is wrong because this table can save useful data for the user.
> At a minimum, we should warn the user about this, but I think we can just add some number at the end of the name, such as name_table1, name_table_2.
>
> Sorry. I don't understand this part.
> Currently, if the error table name already exists, then the copy will
> fail, an error will be reported.
> I try to first create a table, if no error then the error table will be dropped.
>
> To be honest, first of all, I misunderstood this part of the code. Now I see that it works the way you mentioned.
>
> However, I didn't see if you dealt with cases where we already had a table with the same name as the table error.
> I mean, when is he trying to create for the first time, or will we never be able to face such a problem?
>
> Can you demo the expected behavior?
>
> Unfortunately, I was unable to launch it due to a build issue.
>
Hopefully attached will work.
> 2. I noticed that you are forming a table name using the type of errors that prevent rows from being added during 'copy from' operation.
> I think it would be better to use the name of the source file that was used while 'copy from' was running.
> In addition, there may be several such files, it is also worth considering.
>
> Another column added.
> now it looks like:
>
> SELECT * FROM save_error_csv_error;
> filename | lineno | line
> | field | source | err_message |
> err_detail | errorcode
> ----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
> STDIN | 1 | 2002 232 40 50 60 70
> 80 | NULL | NULL | extra data after last expected column |
> NULL | 22P04
> STDIN | 1 | 2000 230 23
> | d | NULL | missing data for column "d" | NULL
> | 22P04
> STDIN | 1 | z,,""
> | a | z | invalid input syntax for type integer: "z" | NULL
> | 22P02
> STDIN | 2 | \0,,
> | a | \0 | invalid input syntax for type integer: "\0" | NULL
> | 22P02
>
> Yes, I see the "filename" column, and this will solve the problem, but "STDIN" is unclear to me.
please see comment in struct CopyFromStateData:
char *filename; /* filename, or NULL for STDIN */
> */
>
> Maybe we can rewrite it like this:
>
> /* Check, the err_nsp.error_rel table has already existed
> * and if it is, check its column name and data types.
>
refactored.
Attachments:
[application/x-patch] v10-0001-Make-COPY-FROM-more-error-tolerant.patch (41.8K, ../../CACJufxHjfcetqip_3RLyvMQSB-SkvxwXkCP3VUN8QsKMKfjoUA@mail.gmail.com/2-v10-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 2510dc2e2b13c60a5a7e184bf8e55325601d97e0 Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Sun, 10 Dec 2023 09:51:42 +0800
Subject: [PATCH v10 1/1] Make COPY FROM more error tolerant
Currently COPY FROM has 3 types of error while processing the source file.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error will save errors to a table automatically.
We check the table definition via column name and column data type.
if table already exists and meets the criteria then errors will save to that table.
if the table does not exist, then create one.
Only works for COPY FROM, non-BINARY mode.
While copying, if error never happened, error save table will be dropped at the ending of COPY FROM.
If the error saving table already exists, meaning at least once COPY FROM errors has happened,
then all the future errors will be saved to that table.
we save the error to error saving table using SPI, construct a query, then execute the query.
---
contrib/file_fdw/file_fdw.c | 4 +-
doc/src/sgml/ref/copy.sgml | 93 +++++++++++++
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 146 +++++++++++++++++++-
src/backend/commands/copyfromparse.c | 169 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 3 +-
src/include/commands/copyfrom_internal.h | 7 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 135 ++++++++++++++++++
src/test/regress/sql/copy2.sql | 108 +++++++++++++++
12 files changed, 670 insertions(+), 19 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 2189be8a..2d3eb34f 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -751,7 +751,7 @@ fileIterateForeignScan(ForeignScanState *node)
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
found = NextCopyFrom(festate->cstate, econtext,
- slot->tts_values, slot->tts_isnull);
+ slot->tts_values, slot->tts_isnull, NULL);
if (found)
ExecStoreVirtualTuple(slot);
@@ -1183,7 +1183,7 @@ file_acquire_sample_rows(Relation onerel, int elevel,
MemoryContextReset(tupcontext);
MemoryContextSwitchTo(tupcontext);
- found = NextCopyFrom(cstate, NULL, values, nulls);
+ found = NextCopyFrom(cstate, NULL, values, nulls, NULL);
MemoryContextSwitchTo(oldcontext);
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..a6370c42 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,17 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion failure while copying will automatically report error information to a regular table.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -572,6 +584,12 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is spceified and a conversion error occurs while copying, then
+ <productname>PostgreSQL</productname> will create a table to save all the conversion errors. Conversion error
+ include data type conversion failure, extra data or missing data in the source file.
+ </para>
+
</refsect1>
<refsect1>
@@ -962,6 +980,81 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title>Error Save Table </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> spceicfied, all the data type conversion fail while copying will automatically saved in a regular table.
+ <xref linkend="copy-errorsave-table"/> shows the error save table name, data type, and description.
+ </para>
+
+ <table id="copy-errorsave-table">
+
+ <title>COPY ERROR SAVE TABLE </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the input file</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of error occuring line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>field</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field name of the error occuring</entry>
+ </row>
+
+ <row>
+ <entry> <literal>source</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occuring field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code for the copying error</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..90a22431 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -38,6 +38,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -652,10 +653,12 @@ CopyFrom(CopyFromState cstate)
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ StringInfo err_save_buf;
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -952,6 +955,7 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ err_save_buf = makeStringInfo();
for (;;)
{
TupleTableSlot *myslot;
@@ -989,8 +993,54 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
/* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull, err_save_buf))
+ {
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->error_nsp && cstate->error_rel);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%ld rows were skipped because of error."
+ " skipped row saved to table %s.%s",
+ cstate->error_rows_cnt,
+ cstate->error_nsp, cstate->error_rel));
+ }
+ else
+ {
+ StringInfoData querybuf;
+ if (cstate->error_firsttime)
+ {
+ ereport(NOTICE,
+ errmsg("No error happened."
+ "Error holding table %s.%s will be droped",
+ cstate->error_nsp, cstate->error_rel));
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DROP TABLE IF EXISTS %s.%s CASCADE ",
+ cstate->error_nsp, cstate->error_rel);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+ }
+ else
+ ereport(NOTICE,
+ errmsg("No error happened. "
+ "All the past error holding saved at %s.%s ",
+ cstate->error_nsp, cstate->error_rel));
+ }
+ }
break;
+ }
+
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
ExecStoreVirtualTuple(myslot);
@@ -1444,6 +1494,98 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ char *err_nsp;
+ char error_rel[NAMEDATALEN];
+ StringInfoData querybuf;
+ bool isnull;
+ bool error_table_ok;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ snprintf(error_rel, sizeof(error_rel), "%s",
+ RelationGetRelationName(cstate->rel));
+ strlcat(error_rel,"_error", NAMEDATALEN);
+ err_nsp = get_namespace_name(RelationGetNamespace(cstate->rel));
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the err_nsp.error_rel table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,filename,lineno,line,field,source,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+
+ appendStringInfo(&querybuf,
+ "relname = $$%s$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ error_rel, err_nsp);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ error_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ /* No err_nsp.error_rel table then create it for holding error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.%s (FILENAME TEXT, LINENO BIGINT, LINE TEXT, "
+ "FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT, "
+ "ERR_DETAIL TEXT, ERRORCODE TEXT)",
+ err_nsp,error_rel);
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ cstate->error_firsttime = true;
+ }
+ else if (error_table_ok)
+ /* error save table already exists. Set error_firsttime to false */
+ cstate->error_firsttime = false;
+ else if(!error_table_ok)
+ ereport(ERROR,
+ (errmsg("Error save table %s.%s already exists. "
+ "Cannot use it for COPY FROM error saving",
+ err_nsp, error_rel)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* thses information is necessary, no error then drop err_sp.error_rel table*/
+ cstate->error_rel = pstrdup(error_rel);
+ cstate->error_nsp = err_nsp;
+ }
+ else
+ {
+ /* set to NULL */
+ cstate->error_rel = NULL;
+ cstate->error_nsp = NULL;
+ cstate->escontext = NULL;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..e7b7a816 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -66,10 +66,12 @@
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -852,7 +854,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
*/
bool
NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls)
+ Datum *values, bool *nulls, StringInfo err_save_buf)
{
TupleDesc tupDesc;
AttrNumber num_phys_attrs,
@@ -885,11 +887,48 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ char *errmsg_extra = "extra data after last expected column";
+
+ resetStringInfo(err_save_buf);
+ /* add line buf, etc for line have extra data to error save table*/
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename, lineno,line, "
+ "err_message, errorcode) "
+ "SELECT $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ errmsg_extra,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +940,46 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ char errmsg[128];
+ snprintf(errmsg, sizeof(errmsg),
+ "missing data for column \"%s\"",
+ NameStr(att->attname));
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename,lineno,line, field, "
+ "err_message, errorcode) "
+ "SELECT $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$ ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ NameStr(att->attname), errmsg,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1031,85 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ {
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ }
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char errcode[12];
+ char *err_detail;
+ snprintf(errcode, sizeof(errcode), "%s",
+ unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename, lineno,line,field, "
+ "source, err_message, errorcode,err_detail) "
+ "SELECT $$%s$$, $$%ld$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$, $$%s$$, ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ cstate->cur_lineno, cstate->line_buf.data,
+ cstate->cur_attname, string,
+ cstate->escontext->error_data->message,
+ errcode);
+
+ if (!err_detail)
+ appendStringInfo(err_save_buf, "NULL::text");
+ else
+ appendStringInfo(err_save_buf,"$$%s$$", err_detail);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_execute failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d631ac89..61b5c5b1 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17329,6 +17333,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -17937,6 +17942,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..de47791a 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
@@ -82,7 +83,7 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls);
+ Datum *values, bool *nulls, StringInfo err_save_buf);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..b1c02b2f 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,12 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ int64 error_rows_cnt; /* total number of rows that have errors */
+ const char *error_rel; /* the error row save table name */
+ const char *error_nsp; /* the error row table's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
+ bool error_firsttime; /* first time create error save table */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..1da12b72 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,116 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+NOTICE: No error happened.Error holding table public.save_error_csv_error will be droped
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+ expected_zero
+---------------
+ 0
+(1 row)
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: Error save table public.save_error_csv_error already exists. Cannot use it for COPY FROM error saving
+DROP TABLE save_error_csv_error;
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of error. skipped row saved to table public.save_error_csv_error
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | d | b_null | b_empty
+---+---+------+------+--------+---------
+ 2 | | NULL | NULL | f | t
+(1 row)
+
+SELECT * FROM save_error_csv_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
+ STDIN | 1 | 2002 232 40 50 60 70 80 | NULL | NULL | extra data after last expected column | NULL | 22P04
+ STDIN | 1 | 2000 230 23 | d | NULL | missing data for column "d" | NULL | 22P04
+ STDIN | 1 | z,,"" | a | z | invalid input syntax for type integer: "z" | NULL | 22P02
+ STDIN | 2 | \0,, | a | \0 | invalid input syntax for type integer: "\0" | NULL | 22P02
+(4 rows)
+
+DROP TABLE save_error_csv, save_error_csv_error;
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 8 rows were skipped because of error. skipped row saved to table public.check_ign_err_error
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+NOTICE: No error happened. All the past error holding saved at public.check_ign_err_error
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+--------------------------------------------+-------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ STDIN | 2 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | " | |
+ STDIN | 3 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 4 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ STDIN | 5 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 9 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(14 rows)
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY check_ign_err FROM STDIN WITH (save_error, save_error o...
+ ^
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 4 rows were skipped because of error. skipped row saved to table public.textrange_input_error
+SELECT * FROM textrange_input_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------+-------+----------+-------------------------------------------------------------------+------------------------------------------+-----------
+ STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 2 | (",a),(",",a),()",a) | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | c | a) | malformed range literal: "a)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 4 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +932,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of error. skipped row saved to table public.copy_default_error_save_error
+select count(*) as expect_zero from copy_default_error_save;
+ expect_zero
+-------------
+ 0
+(1 row)
+
+select * from copy_default_error_save_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..3f43ce75 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,98 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+\.
+
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+DROP TABLE save_error_csv_error;
+
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+2002 232 40 50 60 70 80
+\.
+
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+2000 230 23
+\.
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+
+SELECT * FROM save_error_csv_error;
+
+DROP TABLE save_error_csv, save_error_csv_error;
+
+
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+,,,
+\.
+
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a)
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT * FROM textrange_input_error;
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
+
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +701,19 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+select count(*) as expect_zero from copy_default_error_save;
+select * from copy_default_error_save_error;
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-11 14:05 ` Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alena Rybakina @ 2023-12-11 14:05 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi! Thank you for your work. Your patch looks better!
On 10.12.2023 13:32, jian he wrote:
> On Fri, Dec 8, 2023 at 3:09 PM Alena Rybakina<[email protected]> wrote:
>> Thank you for your work. Unfortunately, your code contained errors during the make installation:
>>
>> 'SAVEPOINT' after 'SAVE_ERROR' in unreserved_keyword list is misplaced
>> 'SAVEPOINT' after 'SAVE_ERROR' in bare_label_keyword list is misplaced
>> make[2]: *** [../../../src/Makefile.global:783: gram.c] Error 1
>> make[1]: *** [Makefile:131: parser/gram.h] Error 2
>> make[1]: *** Waiting for unfinished jobs....
>> make: *** [src/Makefile.global:383: submake-generated-headers] Error 2
>>
>> I have ubuntu 22.04 operation system.
>>
>> On 06.12.2023 13:47, jian he wrote:
>>
>> On Tue, Dec 5, 2023 at 6:07 PM Alena Rybakina<[email protected]> wrote:
>>
>> Hi!
>>
>> Thank you for your contribution to this thread.
>>
>>
>> I reviewed it and have a few questions.
>>
>> 1. I have seen that you delete a table before creating it, to which you want to add errors due to a failed "copy from" operation. I think this is wrong because this table can save useful data for the user.
>> At a minimum, we should warn the user about this, but I think we can just add some number at the end of the name, such as name_table1, name_table_2.
>>
>> Sorry. I don't understand this part.
>> Currently, if the error table name already exists, then the copy will
>> fail, an error will be reported.
>> I try to first create a table, if no error then the error table will be dropped.
>>
>> To be honest, first of all, I misunderstood this part of the code. Now I see that it works the way you mentioned.
>>
>> However, I didn't see if you dealt with cases where we already had a table with the same name as the table error.
>> I mean, when is he trying to create for the first time, or will we never be able to face such a problem?
>>
>> Can you demo the expected behavior?
>>
>> Unfortunately, I was unable to launch it due to a build issue.
>>
> Hopefully attached will work.
Yes, thank you! It works fine, and I see that the regression tests have
been passed. 🙂
However, when I ran 'copy from with save_error' operation with simple
csv files (copy_test.csv, copy_test1.csv) for tables test, test1 (how I
created it, I described below):
postgres=# create table test (x int primary key, y int not null);
postgres=# create table test1 (x int, z int, CONSTRAINT fk_x
FOREIGN KEY(x)
REFERENCES test(x));
I did not find a table with saved errors after operation, although I
received a log about it:
postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
save_error
NOTICE: 2 rows were skipped because of error. skipped row saved to
table public.test_error
ERROR: duplicate key value violates unique constraint "test_pkey"
DETAIL: Key (x)=(2) already exists.
CONTEXT: COPY test, line 3
postgres=# select * from public.test_error;
ERROR: relation "public.test_error" does not exist
LINE 1: select * from public.test_error;
postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ','
CSV save_error
NOTICE: 2 rows were skipped because of error. skipped row saved to
table public.test1_error
ERROR: insert or update on table "test1" violates foreign key
constraint "fk_x"
DETAIL: Key (x)=(2) is not present in table "test".
postgres=# select * from public.test1_error;
ERROR: relation "public.test1_error" does not exist
LINE 1: select * from public.test1_error;
Two lines were written correctly in the csv files, therefore they should
have been added to the tables, but they were not added to the tables
test and test1.
If I leave only the correct rows, everything works fine and the rows are
added to the tables.
in copy_test.csv:
2,0
1,1
in copy_test1.csv:
2,0
2,1
1,1
postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
COPY 2
postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ','
CSV save_error
NOTICE: No error happened.Error holding table public.test1_error will
be droped
COPY 3
Maybe I'm launching it the wrong way. If so, let me know about it.
I also notice interesting behavior if the table was previously created
by the user. When I was creating an error_table before the 'copy from'
operation,
I received a message saying that it is impossible to create a table with
the same name (it is shown below) during the 'copy from' operation.
I think you should add information about this in the documentation,
since this seems to be normal behavior to me.
postgres=# CREATE TABLE test_error (LINENO BIGINT, LINE TEXT,
FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT,
ERR_DETAIL TEXT, ERRORCODE TEXT);
CREATE TABLE
postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
save_error
ERROR: Error save table public.test_error already exists. Cannot use it
for COPY FROM error saving
>
>> 2. I noticed that you are forming a table name using the type of errors that prevent rows from being added during 'copy from' operation.
>> I think it would be better to use the name of the source file that was used while 'copy from' was running.
>> In addition, there may be several such files, it is also worth considering.
>>
>> Another column added.
>> now it looks like:
>>
>> SELECT * FROM save_error_csv_error;
>> filename | lineno | line
>> | field | source | err_message |
>> err_detail | errorcode
>> ----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
>> STDIN | 1 | 2002 232 40 50 60 70
>> 80 | NULL | NULL | extra data after last expected column |
>> NULL | 22P04
>> STDIN | 1 | 2000 230 23
>> | d | NULL | missing data for column "d" | NULL
>> | 22P04
>> STDIN | 1 | z,,""
>> | a | z | invalid input syntax for type integer: "z" | NULL
>> | 22P02
>> STDIN | 2 | \0,,
>> | a | \0 | invalid input syntax for type integer: "\0" | NULL
>> | 22P02
>>
>> Yes, I see the "filename" column, and this will solve the problem, but "STDIN" is unclear to me.
> please see comment in struct CopyFromStateData:
> char *filename; /* filename, or NULL for STDIN */
>
Yes, I can see that.
I haven't figured out how to fix it yet either.
>> */
>>
>> Maybe we can rewrite it like this:
>>
>> /* Check, the err_nsp.error_rel table has already existed
>> * and if it is, check its column name and data types.
>>
> refactored.
Fine)
--
Regards,
Alena Rybakina
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/csv] copy_test1.csv (26B, ../../[email protected]/3-copy_test1.csv)
download | inline:
2,0
1,
2,0
2,1
1,1
1,c
c,2
[text/csv] copy_test.csv (23B, ../../[email protected]/4-copy_test.csv)
download | inline:
2,0
2,0
2,1
1,1
c,0
1,c
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
@ 2023-12-12 13:04 ` jian he <[email protected]>
2023-12-14 14:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: jian he @ 2023-12-12 13:04 UTC (permalink / raw)
To: Alena Rybakina <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Dec 11, 2023 at 10:05 PM Alena Rybakina
<[email protected]> wrote:
>
> Hi! Thank you for your work. Your patch looks better!
> Yes, thank you! It works fine, and I see that the regression tests have been passed. 🙂
> However, when I ran 'copy from with save_error' operation with simple csv files (copy_test.csv, copy_test1.csv) for tables test, test1 (how I created it, I described below):
>
> postgres=# create table test (x int primary key, y int not null);
> postgres=# create table test1 (x int, z int, CONSTRAINT fk_x
> FOREIGN KEY(x)
> REFERENCES test(x));
>
> I did not find a table with saved errors after operation, although I received a log about it:
>
> postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV save_error
> NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test_error
> ERROR: duplicate key value violates unique constraint "test_pkey"
> DETAIL: Key (x)=(2) already exists.
> CONTEXT: COPY test, line 3
>
> postgres=# select * from public.test_error;
> ERROR: relation "public.test_error" does not exist
> LINE 1: select * from public.test_error;
>
> postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
> NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test1_error
> ERROR: insert or update on table "test1" violates foreign key constraint "fk_x"
> DETAIL: Key (x)=(2) is not present in table "test".
>
> postgres=# select * from public.test1_error;
> ERROR: relation "public.test1_error" does not exist
> LINE 1: select * from public.test1_error;
>
> Two lines were written correctly in the csv files, therefore they should have been added to the tables, but they were not added to the tables test and test1.
>
> If I leave only the correct rows, everything works fine and the rows are added to the tables.
>
> in copy_test.csv:
>
> 2,0
>
> 1,1
>
> in copy_test1.csv:
>
> 2,0
>
> 2,1
>
> 1,1
>
> postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
> COPY 2
> postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
> NOTICE: No error happened.Error holding table public.test1_error will be droped
> COPY 3
>
> Maybe I'm launching it the wrong way. If so, let me know about it.
looks like the above is about constraints violation while copying.
constraints violation while copying not in the scope of this patch.
Since COPY FROM is very like the INSERT command,
you do want all the valid constraints to check all the copied rows?
but the notice raised by the patch is not right.
So I place the drop error saving table or raise notice logic above
`ExecResetTupleTable(estate->es_tupleTable, false)` in the function
CopyFrom.
>
> I also notice interesting behavior if the table was previously created by the user. When I was creating an error_table before the 'copy from' operation,
> I received a message saying that it is impossible to create a table with the same name (it is shown below) during the 'copy from' operation.
> I think you should add information about this in the documentation, since this seems to be normal behavior to me.
>
doc changed. you may check it.
Attachments:
[text/x-patch] v11-0001-Make-COPY-FROM-more-error-tolerant.patch (44.0K, ../../CACJufxEFXzxjD9oOq3LoVQAy0KH0TJsDS3UnTtecxx-4J0+2NA@mail.gmail.com/2-v11-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 3024bf3b727b728c58dfef41c62d7a93c083b887 Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Tue, 12 Dec 2023 20:58:45 +0800
Subject: [PATCH v11 1/1] Make COPY FROM more error tolerant
Currently COPY FROM has 3 types of error while processing the source file.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error specifier will
save errors to a error saving table automatically.
We check the error saving table definition by column name and column data type.
if table already exists and meets the criteria then errors will save to that table.
if the table does not exist, then create one.
Only works for COPY FROM, non-BINARY mode.
While copying, if error never happened, error saving table will be dropped at the ending of COPY FROM.
If the error saving table exists, meaning at least once COPY FROM errors has happened,
then all the future errors will be saved to that table.
We save the error related meta info to error saving table using SPI,
that is construct a query string, then execute the query.
---
contrib/file_fdw/file_fdw.c | 4 +-
doc/src/sgml/ref/copy.sgml | 100 +++++++++++++-
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 146 +++++++++++++++++++-
src/backend/commands/copyfromparse.c | 169 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 3 +-
src/include/commands/copyfrom_internal.h | 7 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 135 ++++++++++++++++++
src/test/regress/sql/copy2.sql | 108 +++++++++++++++
12 files changed, 676 insertions(+), 20 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 2189be8a..2d3eb34f 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -751,7 +751,7 @@ fileIterateForeignScan(ForeignScanState *node)
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
found = NextCopyFrom(festate->cstate, econtext,
- slot->tts_values, slot->tts_isnull);
+ slot->tts_values, slot->tts_isnull, NULL);
if (found)
ExecStoreVirtualTuple(slot);
@@ -1183,7 +1183,7 @@ file_acquire_sample_rows(Relation onerel, int elevel,
MemoryContextReset(tupcontext);
MemoryContextSwitchTo(tupcontext);
- found = NextCopyFrom(cstate, NULL, values, nulls);
+ found = NextCopyFrom(cstate, NULL, values, nulls, NULL);
MemoryContextSwitchTo(oldcontext);
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..fb303b4f 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,18 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion errors while copying will automatically saved in an Error Saving table and the <command>COPY FROM</command> operation will not be interrupted by conversion errors.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ If this option is omitted, any data type conversion errors will be raised immediately.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -564,6 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
amount to a considerable amount of wasted disk space if the failure
happened well into a large copy operation. You might wish to invoke
<command>VACUUM</command> to recover the wasted space.
+ To continue copying while skip conversion errors in a <command>COPY FROM</command>, you might wish to specify <literal>SAVE_ERROR</literal>.
</para>
<para>
@@ -572,6 +586,16 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is specified and conversion errors occurred while copying, then
+ <productname>PostgreSQL</productname> will first try to create a regular Error Saving table to save all the conversion errors related information.
+ The Error Saving table naming rule is the existing table name concatenated with <literal>_error</literal>.
+ If <productname>PostgreSQL</productname> cannot create the Error Saving table, <command>COPY FROM</command> operation stops, an error is raised.
+ All the future errors while copying to the same table will automatically saved to the same Error Saving table.
+ Conversion errors includes data type conversion failure, extra data or missing data in the source file.
+ Error Saving table detailed description listed in <xref linkend="copy-errorsave-table"/>.
+ </para>
+
</refsect1>
<refsect1>
@@ -588,7 +612,7 @@ COPY <replaceable class="parameter">count</replaceable>
output function, or acceptable to the input function, of each
attribute's data type. The specified null string is used in
place of columns that are null.
- <command>COPY FROM</command> will raise an error if any line of the
+ By default, if <literal>SAVE_ERROR</literal> not specified, <command>COPY FROM</command> will raise an error if any line of the
input file contains more or fewer columns than are expected.
</para>
@@ -962,6 +986,80 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title>Error Save Table </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> specified, all the data type conversion errors while copying will automatically saved in an Error Saving table.
+ <xref linkend="copy-errorsave-table"/> shows the Error Saving table's column name, data type, and description.
+ </para>
+
+ <table id="copy-errorsave-table">
+ <title>Error Saving table description </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the input file</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where the error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>field</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field name of the error occurred</entry>
+ </row>
+
+ <row>
+ <entry> <literal>source</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code for the copying error</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..236d711b 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -38,6 +38,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -652,10 +653,12 @@ CopyFrom(CopyFromState cstate)
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ StringInfo err_save_buf;
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -952,6 +955,7 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ err_save_buf = makeStringInfo();
for (;;)
{
TupleTableSlot *myslot;
@@ -989,9 +993,13 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
/* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull, err_save_buf))
break;
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1297,6 +1305,48 @@ CopyFrom(CopyFromState cstate)
ExecResetTupleTable(estate->es_tupleTable, false);
+ /* drop the error saving table or raise a notice */
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->error_nsp && cstate->error_rel);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%llu rows were skipped because of conversion error."
+ " Skipped rows saved to table %s.%s",
+ (unsigned long long) cstate->error_rows_cnt,
+ cstate->error_nsp, cstate->error_rel));
+ }
+ else
+ {
+ StringInfoData querybuf;
+ if (cstate->error_firsttime)
+ {
+ ereport(NOTICE,
+ errmsg("No conversion error happened. "
+ "Error Saving table %s.%s will be dropped",
+ cstate->error_nsp, cstate->error_rel));
+ initStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "DROP TABLE IF EXISTS %s.%s CASCADE ",
+ cstate->error_nsp, cstate->error_rel);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+ }
+ else
+ ereport(NOTICE,
+ errmsg("No error happened. "
+ "All previouly encountered conversion errors saved at %s.%s",
+ cstate->error_nsp, cstate->error_rel));
+ }
+ }
+
/* Allow the FDW to shut down */
if (target_resultRelInfo->ri_FdwRoutine != NULL &&
target_resultRelInfo->ri_FdwRoutine->EndForeignInsert != NULL)
@@ -1444,6 +1494,98 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ char *err_nsp;
+ char error_rel[NAMEDATALEN];
+ StringInfoData querybuf;
+ bool isnull;
+ bool error_table_ok;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ snprintf(error_rel, sizeof(error_rel), "%s",
+ RelationGetRelationName(cstate->rel));
+ strlcat(error_rel,"_error", NAMEDATALEN);
+ err_nsp = get_namespace_name(RelationGetNamespace(cstate->rel));
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the err_nsp.error_rel table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,filename,lineno,line,field,source,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+
+ appendStringInfo(&querybuf,
+ "relname = $$%s$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ error_rel, err_nsp);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ error_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ /* No err_nsp.error_rel table then create it for holding error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.%s (FILENAME TEXT, LINENO BIGINT, LINE TEXT, "
+ "FIELD TEXT, SOURCE TEXT, ERR_MESSAGE TEXT, "
+ "ERR_DETAIL TEXT, ERRORCODE TEXT)",
+ err_nsp,error_rel);
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ cstate->error_firsttime = true;
+ }
+ else if (error_table_ok)
+ /* error save table already exists. Set error_firsttime to false */
+ cstate->error_firsttime = false;
+ else if(!error_table_ok)
+ ereport(ERROR,
+ (errmsg("Error save table %s.%s already exists. "
+ "Cannot use it for COPY FROM error saving",
+ err_nsp, error_rel)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* thses information is necessary, no error then drop err_sp.error_rel table*/
+ cstate->error_rel = pstrdup(error_rel);
+ cstate->error_nsp = err_nsp;
+ }
+ else
+ {
+ /* set to NULL */
+ cstate->error_rel = NULL;
+ cstate->error_nsp = NULL;
+ cstate->escontext = NULL;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..aa168d3f 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -66,10 +66,12 @@
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -852,7 +854,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
*/
bool
NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls)
+ Datum *values, bool *nulls, StringInfo err_save_buf)
{
TupleDesc tupDesc;
AttrNumber num_phys_attrs,
@@ -885,11 +887,48 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ char *errmsg_extra = "extra data after last expected column";
+
+ resetStringInfo(err_save_buf);
+ /* add line buf, etc for line have extra data to error save table*/
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename, lineno,line, "
+ "err_message, errorcode) "
+ "SELECT $$%s$$, $$%llu$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data, errmsg_extra,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +940,46 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ char errmsg[128];
+ snprintf(errmsg, sizeof(errmsg),
+ "missing data for column \"%s\"",
+ NameStr(att->attname));
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename,lineno,line, field, "
+ "err_message, errorcode) "
+ "SELECT $$%s$$, $$%llu$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$ ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data, NameStr(att->attname), errmsg,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1031,85 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ {
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ }
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char errcode[12];
+ char *err_detail;
+ snprintf(errcode, sizeof(errcode), "%s",
+ unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.%s(filename, lineno,line,field, "
+ "source, err_message, errorcode,err_detail) "
+ "SELECT $$%s$$, $$%llu$$::bigint, $$%s$$, $$%s$$, "
+ "$$%s$$, $$%s$$, $$%s$$, ",
+ cstate->error_nsp, cstate->error_rel,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data, cstate->cur_attname, string,
+ cstate->escontext->error_data->message,
+ errcode);
+
+ if (!err_detail)
+ appendStringInfo(err_save_buf, "NULL::text");
+ else
+ appendStringInfo(err_save_buf,"$$%s$$", err_detail);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_execute failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index f16bbd3c..3a616ab5 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17346,6 +17350,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -17954,6 +17959,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..de47791a 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
@@ -82,7 +83,7 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls);
+ Datum *values, bool *nulls, StringInfo err_save_buf);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..dd41fcaa 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,12 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 error_rows_cnt; /* total number of rows that have errors */
+ const char *error_rel; /* the error row save table name */
+ const char *error_nsp; /* the error row table's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
+ bool error_firsttime; /* first time create error save table */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..aa1398d7 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,116 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+NOTICE: No conversion error happened. Error Saving table public.save_error_csv_error will be dropped
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+ expected_zero
+---------------
+ 0
+(1 row)
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: Error save table public.save_error_csv_error already exists. Cannot use it for COPY FROM error saving
+DROP TABLE save_error_csv_error;
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of conversion error. Skipped rows saved to table public.save_error_csv_error
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+NOTICE: 1 rows were skipped because of conversion error. Skipped rows saved to table public.save_error_csv_error
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table public.save_error_csv_error
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | d | b_null | b_empty
+---+---+------+------+--------+---------
+ 2 | | NULL | NULL | f | t
+(1 row)
+
+SELECT * FROM save_error_csv_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------------------------+-------+--------+---------------------------------------------+------------+-----------
+ STDIN | 1 | 2002 232 40 50 60 70 80 | NULL | NULL | extra data after last expected column | NULL | 22P04
+ STDIN | 1 | 2000 230 23 | d | NULL | missing data for column "d" | NULL | 22P04
+ STDIN | 1 | z,,"" | a | z | invalid input syntax for type integer: "z" | NULL | 22P02
+ STDIN | 2 | \0,, | a | \0 | invalid input syntax for type integer: "\0" | NULL | 22P02
+(4 rows)
+
+DROP TABLE save_error_csv, save_error_csv_error;
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 8 rows were skipped because of conversion error. Skipped rows saved to table public.check_ign_err_error
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+NOTICE: No error happened. All previouly encountered conversion errors saved at public.check_ign_err_error
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+--------------------------------------------+-------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ STDIN | 2 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | " | |
+ STDIN | 3 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 4 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ STDIN | 5 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ STDIN | 6 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ STDIN | 7 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ STDIN | 8 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ STDIN | 9 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ STDIN | 9 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(14 rows)
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY check_ign_err FROM STDIN WITH (save_error, save_error o...
+ ^
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 4 rows were skipped because of conversion error. Skipped rows saved to table public.textrange_input_error
+SELECT * FROM textrange_input_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------+-------+----------+-------------------------------------------------------------------+------------------------------------------+-----------
+ STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 2 | (",a),(",",a),()",a) | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 2 | (",a),(",",a),()",a) | c | a) | malformed range literal: "a)" | Missing left parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ STDIN | 3 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ STDIN | 4 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ STDIN | 4 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +932,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of conversion error. Skipped rows saved to table public.copy_default_error_save_error
+select count(*) as expect_zero from copy_default_error_save;
+ expect_zero
+-------------
+ 0
+(1 row)
+
+select * from copy_default_error_save_error;
+ filename | lineno | line | field | source | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..3f43ce75 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,98 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+
+--- copy success, error save table will be dropped automatically.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+\.
+
+--error TABLE should already droppped.
+select count(*) as expected_zero from pg_class where relname = 'save_error_csv_error';
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+create table save_error_csv_error();
+--should fail. since table save_error_csv_error already exists.
+--error save table naming logic = copy destination tablename + "_error"
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+DROP TABLE save_error_csv_error;
+
+-- save error with extra data
+COPY save_error_csv from stdin(save_error);
+2002 232 40 50 60 70 80
+\.
+
+-- save error with missing data for column
+COPY save_error_csv from stdin(save_error);
+2000 230 23
+\.
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+
+SELECT * FROM save_error_csv_error;
+
+DROP TABLE save_error_csv, save_error_csv_error;
+
+
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+--special case. will work,but the error TABLE should not DROP.
+COPY check_ign_err FROM STDIN WITH (save_error, format csv, FORCE_NULL *);
+,,,
+\.
+
+--expect error TABLE exists
+SELECT * FROM check_ign_err_error;
+
+-- redundant options not allowed.
+COPY check_ign_err FROM STDIN WITH (save_error, save_error off);
+
+DROP TABLE check_ign_err CASCADE;
+DROP TABLE IF EXISTS check_ign_err_error CASCADE;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+CREATE TABLE textrange_input(a textrange, b textrange, c textrange);
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a)
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT * FROM textrange_input_error;
+DROP TABLE textrange_input;
+DROP TABLE textrange_input_error;
+
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +701,19 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+select count(*) as expect_zero from copy_default_error_save;
+select * from copy_default_error_save_error;
+drop table copy_default_error_save_error,copy_default_error_save;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-14 14:48 ` Alena Rybakina <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Alena Rybakina @ 2023-12-14 14:48 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On 12.12.2023 16:04, jian he wrote:
> On Mon, Dec 11, 2023 at 10:05 PM Alena Rybakina
> <[email protected]> wrote:
>> Hi! Thank you for your work. Your patch looks better!
>> Yes, thank you! It works fine, and I see that the regression tests have been passed. 🙂
>> However, when I ran 'copy from with save_error' operation with simple csv files (copy_test.csv, copy_test1.csv) for tables test, test1 (how I created it, I described below):
>>
>> postgres=# create table test (x int primary key, y int not null);
>> postgres=# create table test1 (x int, z int, CONSTRAINT fk_x
>> FOREIGN KEY(x)
>> REFERENCES test(x));
>>
>> I did not find a table with saved errors after operation, although I received a log about it:
>>
>> postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV save_error
>> NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test_error
>> ERROR: duplicate key value violates unique constraint "test_pkey"
>> DETAIL: Key (x)=(2) already exists.
>> CONTEXT: COPY test, line 3
>>
>> postgres=# select * from public.test_error;
>> ERROR: relation "public.test_error" does not exist
>> LINE 1: select * from public.test_error;
>>
>> postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
>> NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test1_error
>> ERROR: insert or update on table "test1" violates foreign key constraint "fk_x"
>> DETAIL: Key (x)=(2) is not present in table "test".
>>
>> postgres=# select * from public.test1_error;
>> ERROR: relation "public.test1_error" does not exist
>> LINE 1: select * from public.test1_error;
>>
>> Two lines were written correctly in the csv files, therefore they should have been added to the tables, but they were not added to the tables test and test1.
>>
>> If I leave only the correct rows, everything works fine and the rows are added to the tables.
>>
>> in copy_test.csv:
>>
>> 2,0
>>
>> 1,1
>>
>> in copy_test1.csv:
>>
>> 2,0
>>
>> 2,1
>>
>> 1,1
>>
>> postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
>> COPY 2
>> postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
>> NOTICE: No error happened.Error holding table public.test1_error will be droped
>> COPY 3
>>
>> Maybe I'm launching it the wrong way. If so, let me know about it.
> looks like the above is about constraints violation while copying.
> constraints violation while copying not in the scope of this patch.
>
> Since COPY FROM is very like the INSERT command,
> you do want all the valid constraints to check all the copied rows?
No, I think it will be too much.
> but the notice raised by the patch is not right.
> So I place the drop error saving table or raise notice logic above
> `ExecResetTupleTable(estate->es_tupleTable, false)` in the function
> CopyFrom.
Yes, I see it and agree with you.
>> I also notice interesting behavior if the table was previously created by the user. When I was creating an error_table before the 'copy from' operation,
>> I received a message saying that it is impossible to create a table with the same name (it is shown below) during the 'copy from' operation.
>> I think you should add information about this in the documentation, since this seems to be normal behavior to me.
>>
> doc changed. you may check it.
Yes, I saw it. Thank you.
--
Regards,
Alena Rybakina
Postgres Professional:http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-14 20:48 ` Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 02:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
1 sibling, 2 replies; 117+ messages in thread
From: Masahiko Sawada @ 2023-12-14 20:48 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi,
On Tue, Dec 12, 2023 at 10:04 PM jian he <[email protected]> wrote:
>
> On Mon, Dec 11, 2023 at 10:05 PM Alena Rybakina
> <[email protected]> wrote:
> >
> > Hi! Thank you for your work. Your patch looks better!
> > Yes, thank you! It works fine, and I see that the regression tests have been passed. 🙂
> > However, when I ran 'copy from with save_error' operation with simple csv files (copy_test.csv, copy_test1.csv) for tables test, test1 (how I created it, I described below):
> >
> > postgres=# create table test (x int primary key, y int not null);
> > postgres=# create table test1 (x int, z int, CONSTRAINT fk_x
> > FOREIGN KEY(x)
> > REFERENCES test(x));
> >
> > I did not find a table with saved errors after operation, although I received a log about it:
> >
> > postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV save_error
> > NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test_error
> > ERROR: duplicate key value violates unique constraint "test_pkey"
> > DETAIL: Key (x)=(2) already exists.
> > CONTEXT: COPY test, line 3
> >
> > postgres=# select * from public.test_error;
> > ERROR: relation "public.test_error" does not exist
> > LINE 1: select * from public.test_error;
> >
> > postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
> > NOTICE: 2 rows were skipped because of error. skipped row saved to table public.test1_error
> > ERROR: insert or update on table "test1" violates foreign key constraint "fk_x"
> > DETAIL: Key (x)=(2) is not present in table "test".
> >
> > postgres=# select * from public.test1_error;
> > ERROR: relation "public.test1_error" does not exist
> > LINE 1: select * from public.test1_error;
> >
> > Two lines were written correctly in the csv files, therefore they should have been added to the tables, but they were not added to the tables test and test1.
> >
> > If I leave only the correct rows, everything works fine and the rows are added to the tables.
> >
> > in copy_test.csv:
> >
> > 2,0
> >
> > 1,1
> >
> > in copy_test1.csv:
> >
> > 2,0
> >
> > 2,1
> >
> > 1,1
> >
> > postgres=# \copy test from '/home/alena/copy_test.csv' DELIMITER ',' CSV
> > COPY 2
> > postgres=# \copy test1 from '/home/alena/copy_test1.csv' DELIMITER ',' CSV save_error
> > NOTICE: No error happened.Error holding table public.test1_error will be droped
> > COPY 3
> >
> > Maybe I'm launching it the wrong way. If so, let me know about it.
>
> looks like the above is about constraints violation while copying.
> constraints violation while copying not in the scope of this patch.
>
> Since COPY FROM is very like the INSERT command,
> you do want all the valid constraints to check all the copied rows?
>
> but the notice raised by the patch is not right.
> So I place the drop error saving table or raise notice logic above
> `ExecResetTupleTable(estate->es_tupleTable, false)` in the function
> CopyFrom.
>
> >
> > I also notice interesting behavior if the table was previously created by the user. When I was creating an error_table before the 'copy from' operation,
> > I received a message saying that it is impossible to create a table with the same name (it is shown below) during the 'copy from' operation.
> > I think you should add information about this in the documentation, since this seems to be normal behavior to me.
> >
>
> doc changed. you may check it.
I've read this thread and the latest patch. IIUC with SAVE_ERROR
option, COPY FROM creates an error table for the target table and
writes error information there.
While I agree that the final shape of this feature would be something
like that design, I'm concerned some features are missing in order to
make this feature useful in practice. For instance, error logs are
inserted to error tables without bounds, meaning that users who want
to tolerate errors during COPY FROM will have to truncate or drop the
error tables periodically, or the database will grow with error logs
without limit. Ideally such maintenance work should be done by the
database. There might be some users who want to log such conversion
errors in server logs to avoid such maintenance work. I think we
should provide an option for where to write, at least. Also, since the
error tables are normal user tables internally, error logs are also
replicated to subscribers if there is a publication FOR ALL TABLES,
unlike system catalogs. I think some users would not like such
behavior.
Looking at SAVE_ERROR feature closely, I think it consists of two
separate features. That is, it enables COPY FROM to load data while
(1) tolerating errors and (2) logging errors to somewhere (i.e., an
error table). If we implement only (1), it would be like COPY FROM
tolerate errors infinitely and log errors to /dev/null. The user
cannot see the error details but I guess it could still help some
cases as Andres mentioned[1] (it might be a good idea to send the
number of rows successfully loaded in a NOTICE message if some rows
could not be loaded). Then with (2), COPY FROM can log error
information to somewhere such as tables and server logs and the user
can select it. So I'm thinking we may be able to implement this
feature incrementally. The first step would be something like an
option to ignore all errors or an option to specify the maximum number
of errors to tolerate before raising an ERROR. The second step would
be to support logging destinations such as server logs and tables.
Regards,
[1] https://www.postgresql.org/message-id/20231109002600.fuihn34bjqqgmbjm%40awork3.anarazel.de
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2023-12-18 00:15 ` jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-19 00:28 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
1 sibling, 2 replies; 117+ messages in thread
From: jian he @ 2023-12-18 00:15 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada <[email protected]> wrote:
>
> Hi,
>
> I've read this thread and the latest patch. IIUC with SAVE_ERROR
> option, COPY FROM creates an error table for the target table and
> writes error information there.
>
> While I agree that the final shape of this feature would be something
> like that design, I'm concerned some features are missing in order to
> make this feature useful in practice. For instance, error logs are
> inserted to error tables without bounds, meaning that users who want
> to tolerate errors during COPY FROM will have to truncate or drop the
> error tables periodically, or the database will grow with error logs
> without limit. Ideally such maintenance work should be done by the
> database. There might be some users who want to log such conversion
> errors in server logs to avoid such maintenance work. I think we
> should provide an option for where to write, at least. Also, since the
> error tables are normal user tables internally, error logs are also
> replicated to subscribers if there is a publication FOR ALL TABLES,
> unlike system catalogs. I think some users would not like such
> behavior.
save the error metadata to system catalogs would be more expensive,
please see below explanation.
I have no knowledge of publications.
but i feel there is a feature request: publication FOR ALL TABLES
exclude regex_pattern.
Anyway, that would be another topic.
> Looking at SAVE_ERROR feature closely, I think it consists of two
> separate features. That is, it enables COPY FROM to load data while
> (1) tolerating errors and (2) logging errors to somewhere (i.e., an
> error table). If we implement only (1), it would be like COPY FROM
> tolerate errors infinitely and log errors to /dev/null. The user
> cannot see the error details but I guess it could still help some
> cases as Andres mentioned[1] (it might be a good idea to send the
> number of rows successfully loaded in a NOTICE message if some rows
> could not be loaded). Then with (2), COPY FROM can log error
> information to somewhere such as tables and server logs and the user
> can select it. So I'm thinking we may be able to implement this
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
> be to support logging destinations such as server logs and tables.
>
> Regards,
>
> [1] https://www.postgresql.org/message-id/20231109002600.fuihn34bjqqgmbjm%40awork3.anarazel.de
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
I don't think "specify the maximum number of errors to tolerate
before raising an ERROR." is very useful....
QUOTE from [1]
MAXERROR [AS] error_count
If the load returns the error_count number of errors or greater, the
load fails. If the load returns fewer errors, it continues and returns
an INFO message that states the number of rows that could not be
loaded. Use this parameter to allow loads to continue when certain
rows fail to load into the table because of formatting errors or other
inconsistencies in the data.
Set this value to 0 or 1 if you want the load to fail as soon as the
first error occurs. The AS keyword is optional. The MAXERROR default
value is 0 and the limit is 100000.
The actual number of errors reported might be greater than the
specified MAXERROR because of the parallel nature of Amazon Redshift.
If any node in the Amazon Redshift cluster detects that MAXERROR has
been exceeded, each node reports all of the errors it has encountered.
END OF QUOTE
option MAXERROR error_count. iiuc, it fails while validating line
error_count + 1, else it raises a notice, tells you how many rows have
errors.
* case when error_count is small, and the copy fails, it only tells
you that at least the error_count line has malformed data. but what if
the actual malformed rows are very big. In this case, this failure
error message is not that helpful.
* case when error_count is very big, and the copy does not fail. then
the actual malformed data rows are very big (still less than
error_count). but there is no error report, you don't know which line
has an error.
Either way, if the file has a large portion of malformed rows, then
the MAXERROR option does not make sense.
so maybe we don't need a threshold for tolerating errors.
however, we can have an option, not actually copy to the table, but
only validate, similar to NOLOAD in [1]
why we save the error:
* if only a small portion of malformed rows then saving the error
metadata would be cheap.
* if a large portion of malformed rows then copy will be slow but we
saved the error metadata. Now you can fix it based on this error
metadata.
I think saving errors to a regular table or text file seems sane, but
not to a catalog table.
* for a text file with M rows, N fields, contrived corner case would
be (M-2) * N errors, the last 2 rows have the duplicate keys, violate
primary key constraint. In this case, we first insert (M-2) * N rows
to the catalog table then because of errors we undo it.
I think it will be expensive.
* error meta info is not as important as other pg_catalog tables.
log format is quite verbose, save_error to log seems not so good, I guess.
I suppose we can specify an ERRORFILE directory. similar
implementation [2], demo in [3]
it will generate 2 files, one file shows the malform line content as
is, another file shows the error info.
Let's assume we save the error info to a table:
Since the previous thread says one copy operation may create one error
table is not a good idea, looking back, I agree.
Similar to [4]
I come with the following logic/ideas:
* save_error table name be COPY_ERRORS, shema be the same as copy from
destination table.
* one COPY_ERRORS table saves all COPY FROM generated error metadata
* if save_error specified, before do COPY FROM, first check if the
table COPY_ERRORS
exists,
if not then create one? Or raise an error saying that COPY_ERRORS does
not exist, cannot save_error?
* COPY_ERRORS table owner be current database owner?
* Only the table owner is allowed to INSERT/DELETE/UPDATE, others are
not allowed to INSERT/DELETE/UPDATE.
while doing copy error happened, record the userid, then switch
COPY_ERRORS owner execute the insert command
* the user who is doing COPY FROM operation is allowed solely to view
(select) the errored row they generated.
COPY_ERRORS table would be:
userid oid /* the user who is doing this operation */
error_time timestamptz /* when this error
happened. not 100% sure this column is needed */
filename text /* the copy from source */
table_name text /* the copy from destination */
lineno bigint /* the error line number */
line text /* the whole line raw content */
colname text -- Field with the error.
raw_field_value text --- The value for the field that leads to the error.
err_message text -- same as ErrorData->message
err_detail text --same as ErrorData->detail
errorcode text --transformed errcode, example "22P02"
[1] https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-load.html
[2] https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver1...
[3] https://www.sqlshack.com/working-with-line-numbers-and-errors-using-bulk-insert/
[4] https://docs.aws.amazon.com/redshift/latest/dg/r_STL_LOAD_ERRORS.html
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-18 05:09 ` torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: torikoshia @ 2023-12-18 05:09 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi,
> save the error metadata to system catalogs would be more expensive,
> please see below explanation.
> I have no knowledge of publications.
> but i feel there is a feature request: publication FOR ALL TABLES
> exclude regex_pattern.
> Anyway, that would be another topic.
I think saving error metadata to system catalog is not a good idea, too.
And I believe Sawada-san just pointed out missing features and did not
suggested that we use system catalog.
> I don't think "specify the maximum number of errors to tolerate
> before raising an ERROR." is very useful....
That may be so.
I imagine it's useful in some use case since some loading tools have
such options.
Anyway I agree it's not necessary for initial patch as mentioned in [1].
> I suppose we can specify an ERRORFILE directory. similar
> implementation [2], demo in [3]
> it will generate 2 files, one file shows the malform line content as
> is, another file shows the error info.
That may be a good option when considering "(2) logging errors to
somewhere".
What do you think about the proposal to develop these features in
incrementally?
On 2023-12-15 05:48, Masahiko Sawada wrote:
> So I'm thinking we may be able to implement this
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
> be to support logging destinations such as server logs and tables.
[1]
https://www.postgresql.org/message-id/752672.1699474336%40sss.pgh.pa.us
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2023-12-18 07:41 ` jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-18 07:41 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Dec 18, 2023 at 1:09 PM torikoshia <[email protected]> wrote:
>
> Hi,
>
> > save the error metadata to system catalogs would be more expensive,
> > please see below explanation.
> > I have no knowledge of publications.
> > but i feel there is a feature request: publication FOR ALL TABLES
> > exclude regex_pattern.
> > Anyway, that would be another topic.
>
> I think saving error metadata to system catalog is not a good idea, too.
> And I believe Sawada-san just pointed out missing features and did not
> suggested that we use system catalog.
>
> > I don't think "specify the maximum number of errors to tolerate
> > before raising an ERROR." is very useful....
>
> That may be so.
> I imagine it's useful in some use case since some loading tools have
> such options.
> Anyway I agree it's not necessary for initial patch as mentioned in [1].
>
> > I suppose we can specify an ERRORFILE directory. similar
> > implementation [2], demo in [3]
> > it will generate 2 files, one file shows the malform line content as
> > is, another file shows the error info.
>
> That may be a good option when considering "(2) logging errors to
> somewhere".
>
> What do you think about the proposal to develop these features in
> incrementally?
>
I am more with tom's idea [1], that is when errors happen (data type
conversion only), do not fail, AND we save the error to a table. I
guess we can implement this logic together, only with a new COPY
option.
imagine a case (it's not that contrived, imho), while conversion from
text to table's int, postgres isspace is different from the source
text file's isspace logic.
then all the lines are malformed. If we just say on error continue and
not save error meta info, the user is still confused which field has
the wrong data, then the user will probably try to incrementally test
which field contains malformed data.
Since we need to save the error somewhere.
Everyone has the privilege to INSERT can do COPY.
I think we also need to handle the access privilege also.
So like I mentioned above, one copy_error error table hub, then
everyone can view/select their own copy failure record.
but save to a server text file/directory, not easy for an INSERT
privilege user to see these files, I think.
similarly not easy to see these failed records in log for limited privilege.
if someone wants to fail at maxerror rows, they can do it, since we
will count how many rows failed.
even though I didn't get it.
[1] https://www.postgresql.org/message-id/900123.1699488001%40sss.pgh.pa.us
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-19 01:13 ` Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2023-12-19 01:13 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Dec 18, 2023 at 4:41 PM jian he <[email protected]> wrote:
>
> On Mon, Dec 18, 2023 at 1:09 PM torikoshia <[email protected]> wrote:
> >
> > Hi,
> >
> > > save the error metadata to system catalogs would be more expensive,
> > > please see below explanation.
> > > I have no knowledge of publications.
> > > but i feel there is a feature request: publication FOR ALL TABLES
> > > exclude regex_pattern.
> > > Anyway, that would be another topic.
> >
> > I think saving error metadata to system catalog is not a good idea, too.
> > And I believe Sawada-san just pointed out missing features and did not
> > suggested that we use system catalog.
> >
> > > I don't think "specify the maximum number of errors to tolerate
> > > before raising an ERROR." is very useful....
> >
> > That may be so.
> > I imagine it's useful in some use case since some loading tools have
> > such options.
> > Anyway I agree it's not necessary for initial patch as mentioned in [1].
> >
> > > I suppose we can specify an ERRORFILE directory. similar
> > > implementation [2], demo in [3]
> > > it will generate 2 files, one file shows the malform line content as
> > > is, another file shows the error info.
> >
> > That may be a good option when considering "(2) logging errors to
> > somewhere".
> >
> > What do you think about the proposal to develop these features in
> > incrementally?
> >
>
> I am more with tom's idea [1], that is when errors happen (data type
> conversion only), do not fail, AND we save the error to a table. I
> guess we can implement this logic together, only with a new COPY
> option.
If we want only such a feature we need to implement it together (the
patch could be split, though). But if some parts of the feature are
useful for users as well, I'd recommend implementing it incrementally.
That way, the patches can get small and it would be easy for reviewers
and committers to review/commit them.
>
> imagine a case (it's not that contrived, imho), while conversion from
> text to table's int, postgres isspace is different from the source
> text file's isspace logic.
> then all the lines are malformed. If we just say on error continue and
> not save error meta info, the user is still confused which field has
> the wrong data, then the user will probably try to incrementally test
> which field contains malformed data.
>
> Since we need to save the error somewhere.
> Everyone has the privilege to INSERT can do COPY.
> I think we also need to handle the access privilege also.
> So like I mentioned above, one copy_error error table hub, then
> everyone can view/select their own copy failure record.
The error table hub idea is still unclear to me. I assume that there
are error tables at least on each database. And an error table can
have error data that happened during COPY FROM, including malformed
lines. Do the error tables grow without bounds and the users have to
delete rows at some point? If so, who can do that? How can we achieve
that the users can see only errored rows they generated? And the issue
with logical replication also needs to be resolved. Anyway, if we go
this direction, we need to discuss the overall design.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2023-12-20 04:07 ` jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-20 04:07 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Tue, Dec 19, 2023 at 9:14 AM Masahiko Sawada <[email protected]> wrote:
>
>
> The error table hub idea is still unclear to me. I assume that there
> are error tables at least on each database. And an error table can
> have error data that happened during COPY FROM, including malformed
> lines. Do the error tables grow without bounds and the users have to
> delete rows at some point? If so, who can do that? How can we achieve
> that the users can see only errored rows they generated? And the issue
> with logical replication also needs to be resolved. Anyway, if we go
> this direction, we need to discuss the overall design.
>
> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com
Please check my latest attached POC.
Main content is to build spi query, execute the spi query, regress
test and regress output.
copy_errors one per schema.
foo.copy_errors will be owned by the schema: foo owner.
if you can insert to a table in that specific schema let's say foo,
then you will get privilege to INSERT/DELETE/SELECT
to foo.copy_errors.
If you are not a superuser, you are only allowed to do
INSERT/DELETE/SELECT on foo.copy_errors rows where USERID =
current_user::regrole::oid.
This is done via row level security.
Since foo.copy_errors is mainly INSERT operations, if copy_errors grow
too much, that means your source file has many errors, it will take a
very long time to finish the whole COPY. maybe we can capture how many
errors encountered in another client.
I don't know how to deal with logic replication. looking for ideas.
Attachments:
[text/x-patch] v12-0001-Make-COPY-FROM-more-error-tolerant.patch (50.6K, ../../CACJufxGFqfqzueVC7GPr0QARYXRHEsgM3MRc43SRzVw8vZc5eQ@mail.gmail.com/2-v12-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 9affaf6d94eb4afe26fc7181e38e53eed14e0216 Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Wed, 20 Dec 2023 11:26:25 +0800
Subject: [PATCH v12 1/1] Make COPY FROM more error tolerant
Currently COPY FROM has 3 types of error while processing the source file.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error specifier will
save errors to table copy_errors for all the copy from operation in the same schema.
We check the existing copy_error table definition by column name and column data type.
if table already exists and meets the criteria then errors will to it
if the table does not exist, then create one.
copy_errors is per schema, it's owned by schema's owner.
for non-superusers, if you can do insert in that schema, then you can insert to copy_errors,
but you are only allowed to select/delet your own rows, which is judged by current_user
with copy_error's userid column. Priviledge restirction is implmented via ROW LEVEL SECURITY.
Only works for COPY FROM, non-BINARY mode.
While copying, if error never happened, error saving table will be dropped at the ending of COPY FROM.
If the error saving table exists, meaning at least once COPY FROM errors has happened,
then all the future errors will be saved to that table.
We save the error related meta info to error saving table using SPI,
that is construct a query string, then execute the query.
---
contrib/file_fdw/file_fdw.c | 4 +-
doc/src/sgml/ref/copy.sgml | 122 ++++++++++++++-
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 168 ++++++++++++++++++++-
src/backend/commands/copyfromparse.c | 179 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 3 +-
src/include/commands/copyfrom_internal.h | 5 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 160 ++++++++++++++++++++
src/test/regress/sql/copy2.sql | 142 ++++++++++++++++++
12 files changed, 787 insertions(+), 20 deletions(-)
diff --git a/contrib/file_fdw/file_fdw.c b/contrib/file_fdw/file_fdw.c
index 2189be8a..2d3eb34f 100644
--- a/contrib/file_fdw/file_fdw.c
+++ b/contrib/file_fdw/file_fdw.c
@@ -751,7 +751,7 @@ fileIterateForeignScan(ForeignScanState *node)
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
found = NextCopyFrom(festate->cstate, econtext,
- slot->tts_values, slot->tts_isnull);
+ slot->tts_values, slot->tts_isnull, NULL);
if (found)
ExecStoreVirtualTuple(slot);
@@ -1183,7 +1183,7 @@ file_acquire_sample_rows(Relation onerel, int elevel,
MemoryContextReset(tupcontext);
MemoryContextSwitchTo(tupcontext);
- found = NextCopyFrom(cstate, NULL, values, nulls);
+ found = NextCopyFrom(cstate, NULL, values, nulls, NULL);
MemoryContextSwitchTo(oldcontext);
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..3dbf70ee 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,18 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion errors while copying will automatically saved in table <literal>COPY_ERRORS</literal> and the <command>COPY FROM</command> operation will not be interrupted by conversion errors.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ If this option is omitted, any data type conversion errors will be raised immediately.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -564,6 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
amount to a considerable amount of wasted disk space if the failure
happened well into a large copy operation. You might wish to invoke
<command>VACUUM</command> to recover the wasted space.
+ To continue copying while skip conversion errors in a <command>COPY FROM</command>, you might wish to specify <literal>SAVE_ERROR</literal>.
</para>
<para>
@@ -572,6 +586,19 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+
+ If the <literal>SAVE_ERROR</literal> option is specified and conversion errors occur while copying,
+ <productname>PostgreSQL</productname> will first check the table <literal>COPY_ERRORS</literal> existence, then save the conversion error related information to it.
+ If it does exist, but the actual table definition cannot use it to save the error information, an error is raised, <command>COPY FROM</command> operation stops.
+ If it does not exist, <productname>PostgreSQL</productname> will try to create it before doing the actual copy operation.
+ The table <literal>COPY_ERRORS</literal> owner is the current schema owner.
+ All the future errors related information generated while copying data to the same schema will automatically be saved to the same <literal>COPY_ERRORS</literal> table.
+ Copy conversion error is privileged information, non-superusers is only allowed to <literal>SELECT</literal>, <literal>DELETE</literal> or <literal>INSERT</literal> their own row in the <literal>COPY_ERRORS</literal> table.
+ Conversion errors include data type conversion failure, extra data or missing data in the source file.
+ <literal>COPY_ERRORS</literal> table detailed description listed in <xref linkend="copy-errors-table"/>.
+
+ </para>
</refsect1>
<refsect1>
@@ -588,7 +615,7 @@ COPY <replaceable class="parameter">count</replaceable>
output function, or acceptable to the input function, of each
attribute's data type. The specified null string is used in
place of columns that are null.
- <command>COPY FROM</command> will raise an error if any line of the
+ By default, if <literal>SAVE_ERROR</literal> not specified, <command>COPY FROM</command> will raise an error if any line of the
input file contains more or fewer columns than are expected.
</para>
@@ -962,6 +989,99 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title> TABLE COPY_ERRORS </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> specified, all the data type conversion errors while copying will automatically saved in <literal>COPY_ERRORS</literal>
+ <xref linkend="copy-errors-table"/> shows <literal>COPY_ERRORS</literal> table's column name, data type, and description.
+ </para>
+
+ <table id="copy-errors-table">
+ <title>Error Saving table description </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>userid</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The user generated the conversion error.
+ Refer <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_authid</literal>, if correspond <structfield>oid</structfield> deleted in <literal>pg_authid</literal>, it becomes stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>copy_destination</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The <command>COPY FROM</command> operation destination table oid.
+ Refer <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_class</literal> if correspond <structfield>oid</structfield> deleted in <literal>pg_class</literal>, it becomes stale.
+
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the input filed</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where the error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>colname</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field where the error occurred</entry>
+ </row>
+
+ <row>
+ <entry> <literal>raw_field_value</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code for the copying error</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..a84080b4 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -29,7 +29,9 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "access/xlog.h"
+#include "catalog/pg_authid.h"
#include "catalog/namespace.h"
+#include "catalog/pg_namespace.h"
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
@@ -38,6 +40,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -52,6 +55,7 @@
#include "utils/portal.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/syscache.h"
/*
* No more than this many tuples per CopyMultiInsertBuffer
@@ -652,10 +656,12 @@ CopyFrom(CopyFromState cstate)
bool has_before_insert_row_trig;
bool has_instead_insert_row_trig;
bool leafpart_use_multi_insert = false;
+ StringInfo err_save_buf;
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -952,6 +958,7 @@ CopyFrom(CopyFromState cstate)
errcallback.previous = error_context_stack;
error_context_stack = &errcallback;
+ err_save_buf = makeStringInfo();
for (;;)
{
TupleTableSlot *myslot;
@@ -989,9 +996,13 @@ CopyFrom(CopyFromState cstate)
ExecClearTuple(myslot);
/* Directly store the values/nulls array in the slot */
- if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
+ if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull, err_save_buf))
break;
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1297,6 +1308,20 @@ CopyFrom(CopyFromState cstate)
ExecResetTupleTable(estate->es_tupleTable, false);
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->copy_errors_nspname);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%llu rows were skipped because of conversion error."
+ " Skipped rows saved to table %s.copy_errors",
+ (unsigned long long) cstate->error_rows_cnt,
+ cstate->copy_errors_nspname));
+ }
+ }
+
/* Allow the FDW to shut down */
if (target_resultRelInfo->ri_FdwRoutine != NULL &&
target_resultRelInfo->ri_FdwRoutine->EndForeignInsert != NULL)
@@ -1444,6 +1469,145 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ StringInfoData querybuf;
+ bool isnull;
+ bool copy_erros_table_ok;
+ Oid nsp_oid;
+ Oid save_userid;
+ Oid ownerId;
+ int save_sec_context;
+ const char *copy_errors_nspname;
+ HeapTuple utup;
+ HeapTuple tuple;
+ const char *rname;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ copy_errors_nspname = get_namespace_name(RelationGetNamespace(cstate->rel));
+ nsp_oid = get_namespace_oid(copy_errors_nspname, false);
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the nsp_oid.COPY_ERRORS table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,userid,copy_destination,filename,lineno, "
+ "line,colname,raw_field_value,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,oid,oid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+ appendStringInfo(&querybuf,
+ "relname = $$copy_errors$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ copy_errors_nspname);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ copy_erros_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+ /*
+ * Switch to the schema owner's userid, so that the COPY_ERRORS table owned by
+ * that user. Also record the current userid.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(save_userid));
+ if (!HeapTupleIsValid(utup))
+ elog(ERROR, "cache lookup failed for role %u", save_userid);
+
+ rname = pstrdup(NameStr(((Form_pg_authid) GETSTRUCT(utup))->rolname));
+ ReleaseSysCache(utup);
+
+ tuple = SearchSysCache1(NAMESPACEOID, ObjectIdGetDatum(nsp_oid));
+ if (!HeapTupleIsValid(utup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("schema with OID %u does not exist", nsp_oid)));
+ ownerId = ((Form_pg_namespace) GETSTRUCT(tuple))->nspowner;
+ ReleaseSysCache(tuple);
+
+ /* not sure the flag is correct */
+ SetUserIdAndSecContext(ownerId,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ /* No copy_errors_nspname.COPY_ERRORS table then create it for holding all the potential error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.COPY_ERRORS( "
+ "USERID OID, COPY_DESTINATION OID, FILENAME TEXT,LINENO BIGINT "
+ ",LINE TEXT, COLNAME text, RAW_FIELD_VALUE TEXT "
+ ",ERR_MESSAGE TEXT, ERR_DETAIL TEXT, ERRORCODE TEXT)", copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE POLICY copyerror ON %s.COPY_ERRORS "
+ "FOR ALL TO PUBLIC USING (USERID = current_user::regrole::oid)",
+ copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "ALTER TABLE %s.COPY_ERRORS ENABLE ROW LEVEL SECURITY", copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ else if(!copy_erros_table_ok)
+ ereport(ERROR,
+ (errmsg("table %s.COPY_ERRORS already exists. "
+ "cannot use it for COPY FROM error saving",
+ copy_errors_nspname)));
+
+ /* grant INSERT/SELECT on copy_errors to the copy operation user now */
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "GRANT SELECT, DELETE, INSERT ON TABLE %s.COPY_ERRORS TO %s", copy_errors_nspname, rname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->copy_errors_nspname = pstrdup(copy_errors_nspname);
+ }
+ else
+ {
+ cstate->copy_errors_nspname = NULL;
+ cstate->escontext = NULL;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..f1a6f9dc 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -66,10 +66,12 @@
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -852,7 +854,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
*/
bool
NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls)
+ Datum *values, bool *nulls, StringInfo err_save_buf)
{
TupleDesc tupDesc;
AttrNumber num_phys_attrs,
@@ -880,16 +882,60 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
int fldct;
int fieldno;
char *string;
+ char *errmsg_extra;
+ Oid save_userid;
+ int save_sec_context;
/* read raw fields in the next line */
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
+ /* we need to get the current userid for the SPI queries */
+ if (cstate->opts.save_error)
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ errmsg_extra = pstrdup("extra data after last expected column");
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.copy_errors(userid, copy_destination, filename,lineno,line, "
+ "err_message, errorcode) "
+ "SELECT %u, %u,$$%s$$, %llu,$$%s$$, $$%s$$, $$%s$$",
+ cstate->copy_errors_nspname,
+ save_userid,
+ cstate->rel->rd_rel->oid,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data,
+ errmsg_extra,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +947,50 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ char errmsg[128];
+ snprintf(errmsg, sizeof(errmsg),
+ "missing data for column \"%s\"",
+ NameStr(att->attname));
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.copy_errors( "
+ "userid,copy_destination,filename, "
+ "lineno,line,COLNAME, err_message, errorcode) "
+ "SELECT %u, %u, $$%s$$, %llu, $$%s$$, $$%s$$, $$%s$$, $$%s$$ ",
+ cstate->copy_errors_nspname,
+ save_userid,
+ cstate->rel->rd_rel->oid,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data,
+ NameStr(att->attname),
+ errmsg,
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_exec failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1042,84 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char *err_detail;
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ resetStringInfo(err_save_buf);
+ appendStringInfo(err_save_buf,
+ "INSERT INTO %s.copy_errors(userid,copy_destination, "
+ "filename, lineno,line,COLNAME, "
+ "raw_field_value, err_message,errorcode, err_detail) "
+ "SELECT %u, %u, $$%s$$, %llu, $$%s$$, $$%s$$, $$%s$$, $$%s$$, $$%s$$, ",
+ cstate->copy_errors_nspname,
+ save_userid,
+ cstate->rel->rd_rel->oid,
+ cstate->filename ? cstate->filename : "STDIN",
+ (unsigned long long) cstate->cur_lineno,
+ cstate->line_buf.data,
+ cstate->cur_attname,
+ string,
+ cstate->escontext->error_data->message,
+ unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+
+ if (!err_detail)
+ appendStringInfo(err_save_buf, "NULL::text");
+ else
+ appendStringInfo(err_save_buf,"$$%s$$", err_detail);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(err_save_buf->data, false, 0) != SPI_OK_INSERT)
+ elog(ERROR, "SPI_execute failed: %s", err_save_buf->data);
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 63f172e1..f42e72aa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17346,6 +17350,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -17954,6 +17959,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..de47791a 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
@@ -82,7 +83,7 @@ extern CopyFromState BeginCopyFrom(ParseState *pstate, Relation rel, Node *where
bool is_program, copy_data_source_cb data_source_cb, List *attnamelist, List *options);
extern void EndCopyFrom(CopyFromState cstate);
extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
- Datum *values, bool *nulls);
+ Datum *values, bool *nulls, StringInfo err_save_buf);
extern bool NextCopyFromRawFields(CopyFromState cstate,
char ***fields, int *nfields);
extern void CopyFromErrorCallback(void *arg);
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..65e34e89 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,10 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 error_rows_cnt; /* total number of rows that have errors */
+ const char *copy_errors_nspname; /* the copy_errors's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..f5a84487 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,142 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY save_error_csv FROM STDIN WITH (save_error, save_error ...
+ ^
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: table public.COPY_ERRORS already exists. cannot use it for COPY FROM error saving
+drop table COPY_ERRORS;
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | d | b_null | b_empty
+---+---+------+------+--------+---------
+ 2 | | NULL | NULL | f | t
+(1 row)
+
+DROP TABLE save_error_csv;
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 10 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+ relname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+---------------+----------+--------+--------------------------------------------+---------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ check_ign_err | STDIN | 1 | 1 {1} 1 1 extra | NULL | NULL | extra data after last expected column | NULL | 22P04
+ check_ign_err | STDIN | 2 | 2 | m | NULL | missing data for column "m" | NULL | 22P04
+ check_ign_err | STDIN | 3 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | | " | |
+ check_ign_err | STDIN | 4 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 5 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ check_ign_err | STDIN | 6 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(16 rows)
+
+DROP TABLE check_ign_err;
+DROP TABLE COPY_ERRORS;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER test_copy_errors1;
+CREATE USER test_copy_errors2;
+CREATE USER test_copy_errors3;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION test_copy_errors3;
+SET LOCAL search_path TO copy_errors_test;
+GRANT USAGE on schema copy_errors_test to test_copy_errors1,test_copy_errors2,test_copy_errors3;
+GRANT CREATE on schema copy_errors_test to test_copy_errors3;
+set role test_copy_errors3;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to test_copy_errors1;
+GRANT insert on textrange_input to test_copy_errors2;
+set role test_copy_errors1;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+---each user is only allowed to see their own rows.
+--based on userid is the same as current_user.
+select count(*) as should_be_zero
+from copy_errors_test.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+join pg_roles pr on pr.oid = ce.userid
+where ce.userid != current_user::regrole::oid;
+ should_be_zero
+----------------
+ 0
+(1 row)
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+ relname | rolname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+-----------------+-------------------+----------+--------+-----------------------+---------+-----------------+-------------------------------------------------------------------+------------------------------------------+-----------
+ textrange_input | test_copy_errors1 | STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | c | a); | malformed range literal: "a);" | Missing left parenthesis or bracket. | 22P02
+(5 rows)
+
+set role test_copy_errors2;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SAVEPOINT s1;
+--current user (non-super user) are not allowed to update)
+update copy_errors_test.copy_errors set userid = 0;
+ERROR: permission denied for table copy_errors
+ROLLBACK to s1;
+--current user (non-super user) are allowed to delete all the record they created.
+delete from copy_errors_test.copy_errors;
+set role test_copy_errors1;
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+ relname | rolname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+-----------------+-------------------+----------+--------+-----------------------+---------+-----------------+-------------------------------------------------------------------+------------------------------------------+-----------
+ textrange_input | test_copy_errors1 | STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | test_copy_errors1 | STDIN | 2 | (",a),(",",a),()",a); | c | a); | malformed range literal: "a);" | Missing left parenthesis or bracket. | 22P02
+(5 rows)
+
+set role test_copy_errors3;
+--owner allowed to drop the table.
+drop table copy_errors;
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +958,27 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save';
+ filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..a4ef06d9 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,126 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT,
+ d TEXT
+);
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+drop table COPY_ERRORS;
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+DROP TABLE save_error_csv;
+
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1 extra
+2
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+
+DROP TABLE check_ign_err;
+DROP TABLE COPY_ERRORS;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+
+CREATE USER test_copy_errors1;
+CREATE USER test_copy_errors2;
+CREATE USER test_copy_errors3;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION test_copy_errors3;
+SET LOCAL search_path TO copy_errors_test;
+
+GRANT USAGE on schema copy_errors_test to test_copy_errors1,test_copy_errors2,test_copy_errors3;
+GRANT CREATE on schema copy_errors_test to test_copy_errors3;
+set role test_copy_errors3;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to test_copy_errors1;
+GRANT insert on textrange_input to test_copy_errors2;
+
+set role test_copy_errors1;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a);
+\.
+
+---each user is only allowed to see their own rows.
+--based on userid is the same as current_user.
+select count(*) as should_be_zero
+from copy_errors_test.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+join pg_roles pr on pr.oid = ce.userid
+where ce.userid != current_user::regrole::oid;
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+
+set role test_copy_errors2;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SAVEPOINT s1;
+--current user (non-super user) are not allowed to update)
+update copy_errors_test.copy_errors set userid = 0;
+ROLLBACK to s1;
+
+--current user (non-super user) are allowed to delete all the record they created.
+delete from copy_errors_test.copy_errors;
+
+set role test_copy_errors1;
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+
+set role test_copy_errors3;
+--owner allowed to drop the table.
+drop table copy_errors;
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +729,25 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save';
+
+drop table copy_default_error_save;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-20 12:26 ` Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2023-12-20 12:26 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Wed, Dec 20, 2023 at 1:07 PM jian he <[email protected]> wrote:
>
> On Tue, Dec 19, 2023 at 9:14 AM Masahiko Sawada <[email protected]> wrote:
> >
> >
> > The error table hub idea is still unclear to me. I assume that there
> > are error tables at least on each database. And an error table can
> > have error data that happened during COPY FROM, including malformed
> > lines. Do the error tables grow without bounds and the users have to
> > delete rows at some point? If so, who can do that? How can we achieve
> > that the users can see only errored rows they generated? And the issue
> > with logical replication also needs to be resolved. Anyway, if we go
> > this direction, we need to discuss the overall design.
> >
> > Regards,
> >
> > --
> > Masahiko Sawada
> > Amazon Web Services: https://aws.amazon.com
>
> Please check my latest attached POC.
> Main content is to build spi query, execute the spi query, regress
> test and regress output.
Why do we need to use SPI? I think we can form heap tuples and insert
them to the error table. Creating the error table also doesn't need to
use SPI.
>
> copy_errors one per schema.
> foo.copy_errors will be owned by the schema: foo owner.
It seems that the error table is created when the SAVE_ERROR is used
for the first time. It probably blocks concurrent COPY FROM commands
with SAVE_ERROR option to different tables if the error table is not
created yet.
>
> if you can insert to a table in that specific schema let's say foo,
> then you will get privilege to INSERT/DELETE/SELECT
> to foo.copy_errors.
> If you are not a superuser, you are only allowed to do
> INSERT/DELETE/SELECT on foo.copy_errors rows where USERID =
> current_user::regrole::oid.
> This is done via row level security.
I don't think it works. If the user is dropped, the user's oid could
be reused for a different user.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2023-12-28 03:57 ` jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2023-12-28 03:57 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Wed, Dec 20, 2023 at 8:27 PM Masahiko Sawada <[email protected]> wrote:
>
>
> Why do we need to use SPI? I think we can form heap tuples and insert
> them to the error table. Creating the error table also doesn't need to
> use SPI.
>
Thanks for pointing it out. I figured out how to form heap tuples and
insert them to the error table.
but I don't know how to create the error table without using SPI.
Please pointer it out.
> >
> > copy_errors one per schema.
> > foo.copy_errors will be owned by the schema: foo owner.
>
> It seems that the error table is created when the SAVE_ERROR is used
> for the first time. It probably blocks concurrent COPY FROM commands
> with SAVE_ERROR option to different tables if the error table is not
> created yet.
>
I don't know how to solve this problem.... Maybe we can document this.
but it will block the COPY FROM immediately.
> >
> > if you can insert to a table in that specific schema let's say foo,
> > then you will get privilege to INSERT/DELETE/SELECT
> > to foo.copy_errors.
> > If you are not a superuser, you are only allowed to do
> > INSERT/DELETE/SELECT on foo.copy_errors rows where USERID =
> > current_user::regrole::oid.
> > This is done via row level security.
>
> I don't think it works. If the user is dropped, the user's oid could
> be reused for a different user.
>
You are right.
so I changed, now the schema owner will be the error table owner.
every error table tuple inserts,
I switch to schema owner, do the insert, then switch back to the
COPY_FROM operation user.
now everyone (except superuser) will need explicit grant to access the
error table.
Attachments:
[text/x-patch] v13-0001-Make-COPY-FROM-more-error-tolerant.patch (46.5K, ../../CACJufxGLS0SYEhBgFgm53a3aqjm_RMfvshbugFNYunu7fuF4Eg@mail.gmail.com/2-v13-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 8c8c266f1dc809ffa0ec9f4262bdd912ed6b758a Mon Sep 17 00:00:00 2001
From: pgaddict <[email protected]>
Date: Wed, 27 Dec 2023 20:15:24 +0800
Subject: [PATCH v13 1/1] Make COPY FROM more error tolerant
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Currently COPY FROM has 3 types of error while processing the source file.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error specifier will
save errors to table copy_errors for all the copy from operation in the same schema.
We check the existing copy_errors table definition by column name and column data type.
if table already exists and meets the criteria then errors metadata will save to copy_errors.
if the table does not exist, then create one.
table copy_errors is per schema-wise, it's owned by the copy from
operation destination schema's owner.
The table owner has full privilege on copy_errors,
other non-superuser need gain privilege to access it.
Only works for COPY FROM, non-BINARY mode.
---
doc/src/sgml/ref/copy.sgml | 121 ++++++++++++-
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 133 +++++++++++++-
src/backend/commands/copyfromparse.c | 217 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 6 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 137 ++++++++++++++
src/test/regress/sql/copy2.sql | 123 +++++++++++++
11 files changed, 746 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..1d0ff0b6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,18 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion errors while copying will automatically saved in table <literal>COPY_ERRORS</literal> and the <command>COPY FROM</command> operation will not be interrupted by conversion errors.
+ This option is not allowed when using <literal>binary</literal> format. Note that this
+ is only supported in current <command>COPY FROM</command> syntax.
+ If this option is omitted, any data type conversion errors will be raised immediately.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -564,6 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
amount to a considerable amount of wasted disk space if the failure
happened well into a large copy operation. You might wish to invoke
<command>VACUUM</command> to recover the wasted space.
+ To continue copying while skip conversion errors in a <command>COPY FROM</command>, you might wish to specify <literal>SAVE_ERROR</literal>.
</para>
<para>
@@ -572,6 +586,18 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is specified and conversion errors occur while copying,
+ <productname>PostgreSQL</productname> will first check the table <literal>COPY_ERRORS</literal> existence, then save the conversion error related information to it.
+ If it does exist, but the actual table definition cannot use it to save the error information, an error is raised, <command>COPY FROM</command> operation stops.
+ If it does not exist, <productname>PostgreSQL</productname> will try to create it before doing the actual copy operation.
+ The table <literal>COPY_ERRORS</literal> owner is the current schema owner.
+ All the future errors related information generated while copying data to the same schema will automatically be saved to the same <literal>COPY_ERRORS</literal> table.
+ Currenly only the owner can read and write data to table <literal>COPY_ERRORS</literal>.
+ Conversion errors include data type conversion failure, extra data or missing data in the source file.
+ <literal>COPY_ERRORS</literal> table detailed description listed in <xref linkend="copy-errors-table"/>.
+
+ </para>
</refsect1>
<refsect1>
@@ -588,7 +614,7 @@ COPY <replaceable class="parameter">count</replaceable>
output function, or acceptable to the input function, of each
attribute's data type. The specified null string is used in
place of columns that are null.
- <command>COPY FROM</command> will raise an error if any line of the
+ By default, if <literal>SAVE_ERROR</literal> not specified, <command>COPY FROM</command> will raise an error if any line of the
input file contains more or fewer columns than are expected.
</para>
@@ -962,6 +988,99 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title> TABLE COPY_ERRORS </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> specified, all the data type conversion errors while copying will automatically saved in <literal>COPY_ERRORS</literal>
+ <xref linkend="copy-errors-table"/> shows <literal>COPY_ERRORS</literal> table's column name, data type, and description.
+ </para>
+
+ <table id="copy-errors-table">
+ <title>Error Saving table description </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>userid</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The user generated the conversion error.
+ Refer <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_authid</literal>, if correspond <structfield>oid</structfield> deleted in <literal>pg_authid</literal>, it becomes stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>copy_destination</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The <command>COPY FROM</command> operation destination table oid.
+ Refer <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_class</literal> if correspond <structfield>oid</structfield> deleted in <literal>pg_class</literal>, it becomes stale.
+
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the input filed</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where the error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>colname</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field where the error occurred</entry>
+ </row>
+
+ <row>
+ <entry> <literal>raw_field_value</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message text </entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code for the copying error</entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..a972ad87 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -29,7 +29,9 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "access/xlog.h"
+#include "catalog/pg_authid.h"
#include "catalog/namespace.h"
+#include "catalog/pg_namespace.h"
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
@@ -38,6 +40,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -52,6 +55,7 @@
#include "utils/portal.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/syscache.h"
/*
* No more than this many tuples per CopyMultiInsertBuffer
@@ -655,7 +659,8 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +997,10 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1297,6 +1306,20 @@ CopyFrom(CopyFromState cstate)
ExecResetTupleTable(estate->es_tupleTable, false);
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->copy_errors_nspname);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%llu rows were skipped because of conversion error."
+ " Skipped rows saved to table %s.copy_errors",
+ (unsigned long long) cstate->error_rows_cnt,
+ cstate->copy_errors_nspname));
+ }
+ }
+
/* Allow the FDW to shut down */
if (target_resultRelInfo->ri_FdwRoutine != NULL &&
target_resultRelInfo->ri_FdwRoutine->EndForeignInsert != NULL)
@@ -1444,6 +1467,114 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ StringInfoData querybuf;
+ bool isnull;
+ bool copy_erros_table_ok;
+ Oid nsp_oid;
+ Oid save_userid;
+ Oid ownerId;
+ int save_sec_context;
+ const char *copy_errors_nspname;
+ HeapTuple tuple;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ copy_errors_nspname = get_namespace_name(RelationGetNamespace(cstate->rel));
+ nsp_oid = get_namespace_oid(copy_errors_nspname, false);
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the nsp_oid.COPY_ERRORS table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,userid,copy_destination,filename,lineno, "
+ "line,colname,raw_field_value,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,oid,oid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+ appendStringInfo(&querybuf,
+ "relname = $$copy_errors$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ copy_errors_nspname);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ copy_erros_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ tuple = SearchSysCache1(NAMESPACEOID, ObjectIdGetDatum(nsp_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("schema with OID %u does not exist", nsp_oid)));
+ ownerId = ((Form_pg_namespace) GETSTRUCT(tuple))->nspowner;
+ ReleaseSysCache(tuple);
+
+ cstate->copy_errors_owner = ownerId;
+
+ /*
+ * Switch to the schema owner's userid, so that the COPY_ERRORS table owned by
+ * that user.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+
+ SetUserIdAndSecContext(ownerId,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ /* No copy_errors_nspname.COPY_ERRORS table then create it for holding all the potential error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.COPY_ERRORS( "
+ "USERID OID, COPY_DESTINATION OID, FILENAME TEXT,LINENO BIGINT "
+ ",LINE TEXT, COLNAME text, RAW_FIELD_VALUE TEXT "
+ ",ERR_MESSAGE TEXT, ERR_DETAIL TEXT, ERRORCODE TEXT)", copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ else if(!copy_erros_table_ok)
+ ereport(ERROR,
+ (errmsg("table %s.COPY_ERRORS already exists. "
+ "cannot use it for COPY FROM error saving",
+ copy_errors_nspname)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->copy_errors_nspname = pstrdup(copy_errors_nspname);
+ }
+ else
+ {
+ cstate->copy_errors_nspname = NULL;
+ cstate->escontext = NULL;
+ cstate->copy_errors_owner = (Oid) 0;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..f0849725 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -58,18 +58,21 @@
*/
#include "postgres.h"
+#include "access/heapam.h"
#include <ctype.h>
#include <unistd.h>
#include <sys/stat.h>
-
+#include <catalog/namespace.h>
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -880,16 +883,85 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
int fldct;
int fieldno;
char *string;
+ char *errmsg_extra;
+ Oid save_userid = InvalidOid;
+ int save_sec_context = -1;
+ HeapTuple copy_errors_tup;
+ Relation copy_errorsrel;
+ TupleDesc copy_errors_tupDesc;
+ Datum t_values[10];
+ bool t_isnull[10];
/* read raw fields in the next line */
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ if (cstate->opts.save_error)
+ {
+ /*
+ * Open the copy_errors relation. we also need current userid for the later heap inserts.
+ *
+ */
+ copy_errorsrel = table_open(RelnameGetRelid("copy_errors"), RowExclusiveLock);
+ copy_errors_tupDesc = copy_errorsrel->rd_att;
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ }
+
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ errmsg_extra = pstrdup("extra data after last expected column");
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(
+ cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(errmsg_extra);
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +973,55 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(
+ psprintf("missing data for column \"%s\"", NameStr(att->attname)));
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1073,91 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char *err_detail;
+ char *err_code;
+ err_code = pstrdup(unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = CStringGetTextDatum(cstate->cur_attname);
+ t_isnull[5] = false;
+ t_values[6] = CStringGetTextDatum(string);
+ t_isnull[6] = false;
+ t_values[7] = CStringGetTextDatum(cstate->escontext->error_data->message);
+ t_isnull[7] = false;
+ t_values[8] = err_detail ? CStringGetTextDatum(err_detail) : (Datum) 0;
+ t_isnull[8] = err_detail ? false: true;
+ t_values[9] = CStringGetTextDatum(err_code);
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
+ if (cstate->opts.save_error)
+ table_close(copy_errorsrel, RowExclusiveLock);
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 63f172e1..f42e72aa 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -755,7 +755,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3448,6 +3448,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17346,6 +17350,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -17954,6 +17959,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..aa560dbb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..2c3b7b42 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,11 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ Oid copy_errors_owner; /* the owner of copy_errors table */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 error_rows_cnt; /* total number of rows that have errors */
+ const char *copy_errors_nspname; /* the copy_errors's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa..d0988a4c 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..a2c6bf5aa 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,118 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY save_error_csv FROM STDIN WITH (save_error, save_error ...
+ ^
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: table public.COPY_ERRORS already exists. cannot use it for COPY FROM error saving
+drop table COPY_ERRORS;
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | b_null | b_empty
+---+---+------+--------+---------
+ 2 | | NULL | f | t
+(1 row)
+
+DROP TABLE save_error_csv;
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 10 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+ relname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+---------------+----------+--------+--------------------------------------------+---------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ check_ign_err | STDIN | 1 | 1 {1} 1 1 extra | NULL | NULL | extra data after last expected column | NULL | 22P04
+ check_ign_err | STDIN | 2 | 2 | NULL | NULL | missing data for column "m" | NULL | 22P04
+ check_ign_err | STDIN | 3 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | | " | |
+ check_ign_err | STDIN | 4 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 5 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ check_ign_err | STDIN | 6 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(16 rows)
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK to s1;
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+ relname | rolname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+-----------------+----------------+----------+--------+----------------------------+---------+-----------------+-------------------------------------------------------------------+------------------------------------------+-----------
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | c | a); | malformed range literal: "a);" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+--owner allowed to drop the table.
+drop table copy_errors;
+--should fail. no priviledge
+select * from public.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +934,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+ filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..a37986df 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,106 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+drop table COPY_ERRORS;
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+DROP TABLE save_error_csv;
+
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1 extra
+2
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a);
+\.
+
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+
+ROLLBACK to s1;
+
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+
+--owner allowed to drop the table.
+drop table copy_errors;
+
+--should fail. no priviledge
+select * from public.copy_errors;
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +709,26 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-04 16:05 ` vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: vignesh C @ 2024-01-04 16:05 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Thu, 28 Dec 2023 at 09:27, jian he <[email protected]> wrote:
>
> On Wed, Dec 20, 2023 at 8:27 PM Masahiko Sawada <[email protected]> wrote:
> >
> >
> > Why do we need to use SPI? I think we can form heap tuples and insert
> > them to the error table. Creating the error table also doesn't need to
> > use SPI.
> >
> Thanks for pointing it out. I figured out how to form heap tuples and
> insert them to the error table.
> but I don't know how to create the error table without using SPI.
> Please pointer it out.
>
> > >
> > > copy_errors one per schema.
> > > foo.copy_errors will be owned by the schema: foo owner.
> >
> > It seems that the error table is created when the SAVE_ERROR is used
> > for the first time. It probably blocks concurrent COPY FROM commands
> > with SAVE_ERROR option to different tables if the error table is not
> > created yet.
> >
> I don't know how to solve this problem.... Maybe we can document this.
> but it will block the COPY FROM immediately.
>
> > >
> > > if you can insert to a table in that specific schema let's say foo,
> > > then you will get privilege to INSERT/DELETE/SELECT
> > > to foo.copy_errors.
> > > If you are not a superuser, you are only allowed to do
> > > INSERT/DELETE/SELECT on foo.copy_errors rows where USERID =
> > > current_user::regrole::oid.
> > > This is done via row level security.
> >
> > I don't think it works. If the user is dropped, the user's oid could
> > be reused for a different user.
> >
>
> You are right.
> so I changed, now the schema owner will be the error table owner.
> every error table tuple inserts,
> I switch to schema owner, do the insert, then switch back to the
> COPY_FROM operation user.
> now everyone (except superuser) will need explicit grant to access the
> error table.
There are some compilation issues reported at [1] for the patch:
[04:04:26.288] copyfromparse.c: In function ‘NextCopyFrom’:
[04:04:26.288] copyfromparse.c:1126:25: error: ‘copy_errors_tupDesc’
may be used uninitialized in this function
[-Werror=maybe-uninitialized]
[04:04:26.288] 1126 | copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
[04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[04:04:26.288] 1127 | t_values,
[04:04:26.288] | ~~~~~~~~~
[04:04:26.288] 1128 | t_isnull);
[04:04:26.288] | ~~~~~~~~~
[04:04:26.288] copyfromparse.c:1160:4: error: ‘copy_errorsrel’ may be
used uninitialized in this function [-Werror=maybe-uninitialized]
[04:04:26.288] 1160 | table_close(copy_errorsrel, RowExclusiveLock);
[04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[1] - https://cirrus-ci.com/task/4785221183209472
Regards,
Vignesh
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
@ 2024-01-05 08:37 ` jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-05 08:37 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Fri, Jan 5, 2024 at 12:05 AM vignesh C <[email protected]> wrote:
>
> On Thu, 28 Dec 2023 at 09:27, jian he <[email protected]> wrote:
> >
> > On Wed, Dec 20, 2023 at 8:27 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > >
> > > Why do we need to use SPI? I think we can form heap tuples and insert
> > > them to the error table. Creating the error table also doesn't need to
> > > use SPI.
> > >
> > Thanks for pointing it out. I figured out how to form heap tuples and
> > insert them to the error table.
> > but I don't know how to create the error table without using SPI.
> > Please pointer it out.
> >
> > > >
> > > > copy_errors one per schema.
> > > > foo.copy_errors will be owned by the schema: foo owner.
> > >
> > > It seems that the error table is created when the SAVE_ERROR is used
> > > for the first time. It probably blocks concurrent COPY FROM commands
> > > with SAVE_ERROR option to different tables if the error table is not
> > > created yet.
> > >
> > I don't know how to solve this problem.... Maybe we can document this.
> > but it will block the COPY FROM immediately.
> >
> > > >
> > > > if you can insert to a table in that specific schema let's say foo,
> > > > then you will get privilege to INSERT/DELETE/SELECT
> > > > to foo.copy_errors.
> > > > If you are not a superuser, you are only allowed to do
> > > > INSERT/DELETE/SELECT on foo.copy_errors rows where USERID =
> > > > current_user::regrole::oid.
> > > > This is done via row level security.
> > >
> > > I don't think it works. If the user is dropped, the user's oid could
> > > be reused for a different user.
> > >
> >
> > You are right.
> > so I changed, now the schema owner will be the error table owner.
> > every error table tuple inserts,
> > I switch to schema owner, do the insert, then switch back to the
> > COPY_FROM operation user.
> > now everyone (except superuser) will need explicit grant to access the
> > error table.
>
> There are some compilation issues reported at [1] for the patch:
> [04:04:26.288] copyfromparse.c: In function ‘NextCopyFrom’:
> [04:04:26.288] copyfromparse.c:1126:25: error: ‘copy_errors_tupDesc’
> may be used uninitialized in this function
> [-Werror=maybe-uninitialized]
> [04:04:26.288] 1126 | copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
> [04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> [04:04:26.288] 1127 | t_values,
> [04:04:26.288] | ~~~~~~~~~
> [04:04:26.288] 1128 | t_isnull);
> [04:04:26.288] | ~~~~~~~~~
> [04:04:26.288] copyfromparse.c:1160:4: error: ‘copy_errorsrel’ may be
> used uninitialized in this function [-Werror=maybe-uninitialized]
> [04:04:26.288] 1160 | table_close(copy_errorsrel, RowExclusiveLock);
> [04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> [1] - https://cirrus-ci.com/task/4785221183209472
>
I fixed this issue, and also improved the doc.
Other implementations have not changed.
Attachments:
[application/x-patch] v14-0001-Make-COPY-FROM-more-error-tolerant.patch (46.6K, ../../CACJufxFY2pL7SMOsu4ViwmoxRTLK3rroTuhNRbhcB9wPr+x_3A@mail.gmail.com/2-v14-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From 99ebe03caa9d50b2cd3cdcd05becccd4b61684e1 Mon Sep 17 00:00:00 2001
From: jian he <[email protected]>
Date: Fri, 5 Jan 2024 16:29:38 +0800
Subject: [PATCH v14 1/1] Make COPY FROM more error tolerant
At present, when processing the source file, COPY FROM may encounter three types of data type conversion errors.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error (boolean) specifier will
save errors to the table copy_errors for all the copy from operation happend in the same schema.
We check the existence of table copy_errors,
we also check the data definition of copy_errors via compare column names and data types.
If copy_errors already exists and meets the criteria then errors metadata will save to it.
If copy_errors does not exist, then create it.
If copy_errors exist, cannot use for saving error, then raise an error.
the table copy_errors is per schema-wise, it's owned by the copy from
operation destination schema's owner.
The table owner has full privilege on copy_errors,
other non-superuser need gain privilege to access it.
---
doc/src/sgml/ref/copy.sgml | 120 ++++++++++++-
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 133 +++++++++++++-
src/backend/commands/copyfromparse.c | 217 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 6 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 137 ++++++++++++++
src/test/regress/sql/copy2.sql | 123 +++++++++++++
11 files changed, 745 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..f6cdf0cf 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,18 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion errors while copying will automatically saved in table <literal>COPY_ERRORS</literal> and the <command>COPY FROM</command> operation will not be interrupted by conversion errors.
+ This option is not allowed when using <literal>binary</literal> format. This option
+ is only supported for <command>COPY FROM</command> syntax.
+ If this option is omitted, any data type conversion errors will be raised immediately.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -564,6 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
amount to a considerable amount of wasted disk space if the failure
happened well into a large copy operation. You might wish to invoke
<command>VACUUM</command> to recover the wasted space.
+ To continue copying while skip conversion errors in a <command>COPY FROM</command>, you might wish to specify <literal>SAVE_ERROR</literal>.
</para>
<para>
@@ -572,6 +586,18 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is specified and conversion errors occur while copying,
+ <productname>PostgreSQL</productname> will first check for the existence of the table <literal>COPY_ERRORS</literal>, then save the conversion error information to it.
+ If it does exist, but the table definition cannot use it to save the error, an error is raised, <command>COPY FROM</command> operation stops.
+ If it does not exist, <productname>PostgreSQL</productname> will try to create it before doing the actual copy operation.
+ The table <literal>COPY_ERRORS</literal> owner is the current <command>COPY FROM</command> operation's schema owner.
+ All the future errors related information generated while copying data to the same schema will automatically be saved to the same <literal>COPY_ERRORS</literal> table.
+ Currenly only the owner can read and write data to <literal>COPY_ERRORS</literal>.
+ Conversion errors include data type conversion failure, extra data or missing data in the source file.
+ <literal>COPY_ERRORS</literal> table detailed description listed in <xref linkend="copy-errors-table"/>.
+
+ </para>
</refsect1>
<refsect1>
@@ -588,7 +614,7 @@ COPY <replaceable class="parameter">count</replaceable>
output function, or acceptable to the input function, of each
attribute's data type. The specified null string is used in
place of columns that are null.
- <command>COPY FROM</command> will raise an error if any line of the
+ By default, if <literal>SAVE_ERROR</literal> not specified, <command>COPY FROM</command> will raise an error if any line of the
input file contains more or fewer columns than are expected.
</para>
@@ -962,6 +988,98 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title> Table COPY_ERRORS </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> specified, all the data type conversion errors while copying will automatically saved in <literal>COPY_ERRORS</literal>.
+ <xref linkend="copy-errors-table"/> shows <literal>COPY_ERRORS</literal> table's column name, data type, and description.
+ </para>
+
+ <table id="copy-errors-table">
+ <title>Error Saving table description </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>userid</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The user generated the conversion error.
+ Refer <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_authid</literal>. If the correspond <structfield>oid</structfield> deleted in <literal>pg_authid</literal>, this value become stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>copy_destination</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The <command>COPY FROM</command> operation destination table oid.
+ Refer <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_class</literal>. If the correspond <structfield>oid</structfield> deleted in <literal>pg_class</literal>, this value become stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the <command>COPY FROM</command> input</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where the error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>colname</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field where the error occurred</entry>
+ </row>
+
+ <row>
+ <entry> <literal>raw_field_value</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..a972ad87 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -29,7 +29,9 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "access/xlog.h"
+#include "catalog/pg_authid.h"
#include "catalog/namespace.h"
+#include "catalog/pg_namespace.h"
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
@@ -38,6 +40,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -52,6 +55,7 @@
#include "utils/portal.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/syscache.h"
/*
* No more than this many tuples per CopyMultiInsertBuffer
@@ -655,7 +659,8 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +997,10 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1297,6 +1306,20 @@ CopyFrom(CopyFromState cstate)
ExecResetTupleTable(estate->es_tupleTable, false);
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->copy_errors_nspname);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%llu rows were skipped because of conversion error."
+ " Skipped rows saved to table %s.copy_errors",
+ (unsigned long long) cstate->error_rows_cnt,
+ cstate->copy_errors_nspname));
+ }
+ }
+
/* Allow the FDW to shut down */
if (target_resultRelInfo->ri_FdwRoutine != NULL &&
target_resultRelInfo->ri_FdwRoutine->EndForeignInsert != NULL)
@@ -1444,6 +1467,114 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ StringInfoData querybuf;
+ bool isnull;
+ bool copy_erros_table_ok;
+ Oid nsp_oid;
+ Oid save_userid;
+ Oid ownerId;
+ int save_sec_context;
+ const char *copy_errors_nspname;
+ HeapTuple tuple;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ copy_errors_nspname = get_namespace_name(RelationGetNamespace(cstate->rel));
+ nsp_oid = get_namespace_oid(copy_errors_nspname, false);
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the nsp_oid.COPY_ERRORS table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,userid,copy_destination,filename,lineno, "
+ "line,colname,raw_field_value,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,oid,oid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+ appendStringInfo(&querybuf,
+ "relname = $$copy_errors$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ copy_errors_nspname);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ copy_erros_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ tuple = SearchSysCache1(NAMESPACEOID, ObjectIdGetDatum(nsp_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("schema with OID %u does not exist", nsp_oid)));
+ ownerId = ((Form_pg_namespace) GETSTRUCT(tuple))->nspowner;
+ ReleaseSysCache(tuple);
+
+ cstate->copy_errors_owner = ownerId;
+
+ /*
+ * Switch to the schema owner's userid, so that the COPY_ERRORS table owned by
+ * that user.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+
+ SetUserIdAndSecContext(ownerId,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ /* No copy_errors_nspname.COPY_ERRORS table then create it for holding all the potential error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.COPY_ERRORS( "
+ "USERID OID, COPY_DESTINATION OID, FILENAME TEXT,LINENO BIGINT "
+ ",LINE TEXT, COLNAME text, RAW_FIELD_VALUE TEXT "
+ ",ERR_MESSAGE TEXT, ERR_DETAIL TEXT, ERRORCODE TEXT)", copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ else if(!copy_erros_table_ok)
+ ereport(ERROR,
+ (errmsg("table %s.COPY_ERRORS already exists. "
+ "cannot use it for COPY FROM error saving",
+ copy_errors_nspname)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->copy_errors_nspname = pstrdup(copy_errors_nspname);
+ }
+ else
+ {
+ cstate->copy_errors_nspname = NULL;
+ cstate->escontext = NULL;
+ cstate->copy_errors_owner = (Oid) 0;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..37f36ea0 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -58,18 +58,21 @@
*/
#include "postgres.h"
+#include "access/heapam.h"
#include <ctype.h>
#include <unistd.h>
#include <sys/stat.h>
-
+#include <catalog/namespace.h>
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -880,16 +883,85 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
int fldct;
int fieldno;
char *string;
+ char *errmsg_extra;
+ Oid save_userid = InvalidOid;
+ int save_sec_context = -1;
+ HeapTuple copy_errors_tup;
+ Relation copy_errorsrel;
+ TupleDesc copy_errors_tupDesc;
+ Datum t_values[10] = {0};
+ bool t_isnull[10] = {0};
/* read raw fields in the next line */
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ if (cstate->opts.save_error)
+ {
+ /*
+ * Open the copy_errors relation. we also need current userid for the later heap inserts.
+ *
+ */
+ copy_errorsrel = table_open(RelnameGetRelid("copy_errors"), RowExclusiveLock);
+ copy_errors_tupDesc = copy_errorsrel->rd_att;
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ }
+
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ errmsg_extra = pstrdup("extra data after last expected column");
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(
+ cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(errmsg_extra);
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +973,55 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(
+ psprintf("missing data for column \"%s\"", NameStr(att->attname)));
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1073,91 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char *err_detail;
+ char *err_code;
+ err_code = pstrdup(unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = CStringGetTextDatum(cstate->cur_attname);
+ t_isnull[5] = false;
+ t_values[6] = CStringGetTextDatum(string);
+ t_isnull[6] = false;
+ t_values[7] = CStringGetTextDatum(cstate->escontext->error_data->message);
+ t_isnull[7] = false;
+ t_values[8] = err_detail ? CStringGetTextDatum(err_detail) : (Datum) 0;
+ t_isnull[8] = err_detail ? false: true;
+ t_values[9] = CStringGetTextDatum(err_code);
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
+ if (cstate->opts.save_error)
+ table_close(copy_errorsrel, RowExclusiveLock);
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4b175ef6..fc69420e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -778,7 +778,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3473,6 +3473,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17768,6 +17772,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -18395,6 +18400,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..aa560dbb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..2c3b7b42 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,11 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ Oid copy_errors_owner; /* the owner of copy_errors table */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 error_rows_cnt; /* total number of rows that have errors */
+ const char *copy_errors_nspname; /* the copy_errors's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f88a6c9a..b6f7ed48 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -390,6 +390,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..a2c6bf5aa 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,118 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY save_error_csv FROM STDIN WITH (save_error, save_error ...
+ ^
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: table public.COPY_ERRORS already exists. cannot use it for COPY FROM error saving
+drop table COPY_ERRORS;
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | b_null | b_empty
+---+---+------+--------+---------
+ 2 | | NULL | f | t
+(1 row)
+
+DROP TABLE save_error_csv;
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 10 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+ relname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+---------------+----------+--------+--------------------------------------------+---------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ check_ign_err | STDIN | 1 | 1 {1} 1 1 extra | NULL | NULL | extra data after last expected column | NULL | 22P04
+ check_ign_err | STDIN | 2 | 2 | NULL | NULL | missing data for column "m" | NULL | 22P04
+ check_ign_err | STDIN | 3 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | | " | |
+ check_ign_err | STDIN | 4 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 5 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ check_ign_err | STDIN | 6 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(16 rows)
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK to s1;
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+ relname | rolname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+-----------------+----------------+----------+--------+----------------------------+---------+-----------------+-------------------------------------------------------------------+------------------------------------------+-----------
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | c | a); | malformed range literal: "a);" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+--owner allowed to drop the table.
+drop table copy_errors;
+--should fail. no priviledge
+select * from public.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +934,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+ filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..a37986df 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,106 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+drop table COPY_ERRORS;
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+DROP TABLE save_error_csv;
+
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1 extra
+2
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a);
+\.
+
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+
+ROLLBACK to s1;
+
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+
+--owner allowed to drop the table.
+drop table copy_errors;
+
+--should fail. no priviledge
+select * from public.copy_errors;
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +709,26 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-06 00:50 ` jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-06 00:50 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; torikoshia <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Fri, Jan 5, 2024 at 4:37 PM jian he <[email protected]> wrote:
>
> > > > be reused for a different user.
> > > >
> > >
> > > You are right.
> > > so I changed, now the schema owner will be the error table owner.
> > > every error table tuple inserts,
> > > I switch to schema owner, do the insert, then switch back to the
> > > COPY_FROM operation user.
> > > now everyone (except superuser) will need explicit grant to access the
> > > error table.
> >
> > There are some compilation issues reported at [1] for the patch:
> > [04:04:26.288] copyfromparse.c: In function ‘NextCopyFrom’:
> > [04:04:26.288] copyfromparse.c:1126:25: error: ‘copy_errors_tupDesc’
> > may be used uninitialized in this function
> > [-Werror=maybe-uninitialized]
> > [04:04:26.288] 1126 | copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
> > [04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > [04:04:26.288] 1127 | t_values,
> > [04:04:26.288] | ~~~~~~~~~
> > [04:04:26.288] 1128 | t_isnull);
> > [04:04:26.288] | ~~~~~~~~~
> > [04:04:26.288] copyfromparse.c:1160:4: error: ‘copy_errorsrel’ may be
> > used uninitialized in this function [-Werror=maybe-uninitialized]
> > [04:04:26.288] 1160 | table_close(copy_errorsrel, RowExclusiveLock);
> > [04:04:26.288] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > [1] - https://cirrus-ci.com/task/4785221183209472
> >
>
> I fixed this issue, and also improved the doc.
> Other implementations have not changed.
bother again.
This time, I used the ci test it again.
now there should be no warning.
Attachments:
[application/x-patch] v15-0001-Make-COPY-FROM-more-error-tolerant.patch (46.6K, ../../CACJufxEkkqnozdnvNMGxVAA94KZaCPkYw_Cx4JKG9ueNaZma_A@mail.gmail.com/2-v15-0001-Make-COPY-FROM-more-error-tolerant.patch)
download | inline diff:
From f033ef4025dbe2012007434dacd4821718443571 Mon Sep 17 00:00:00 2001
From: jian he <[email protected]>
Date: Sat, 6 Jan 2024 02:34:22 +0800
Subject: [PATCH v14 1/1] Make COPY FROM more error tolerant
At present, when processing the source file, COPY FROM may encounter three types of data type conversion errors.
* extra data after last expected column
* missing data for column \"%s\"
* data type conversion error.
Instead of throwing errors while copying, save_error (boolean) specifier will
save errors to the table copy_errors for all the copy from operation happend in the same schema.
We check the existence of table copy_errors,
we also check the data definition of copy_errors via compare column names and data types.
If copy_errors already exists and meets the criteria then errors metadata will save to it.
If copy_errors does not exist, then create it.
If copy_errors exist, cannot use for saving error, then raise an error.
the table copy_errors is per schema-wise, it's owned by the copy from
operation destination schema's owner.
The table owner has full privilege on copy_errors,
other non-superuser need gain privilege to access it.
---
doc/src/sgml/ref/copy.sgml | 120 ++++++++++++-
src/backend/commands/copy.c | 12 ++
src/backend/commands/copyfrom.c | 133 +++++++++++++-
src/backend/commands/copyfromparse.c | 217 +++++++++++++++++++++--
src/backend/parser/gram.y | 8 +-
src/bin/psql/tab-complete.c | 3 +-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 6 +
src/include/parser/kwlist.h | 1 +
src/test/regress/expected/copy2.out | 137 ++++++++++++++
src/test/regress/sql/copy2.sql | 123 +++++++++++++
11 files changed, 745 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c..f6cdf0cf 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -44,6 +44,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
+ SAVE_ERROR [ <replaceable class="parameter">boolean</replaceable> ]
</synopsis>
</refsynopsisdiv>
@@ -411,6 +412,18 @@ WHERE <replaceable class="parameter">condition</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR</literal></term>
+ <listitem>
+ <para>
+ Specifies that any data conversion errors while copying will automatically saved in table <literal>COPY_ERRORS</literal> and the <command>COPY FROM</command> operation will not be interrupted by conversion errors.
+ This option is not allowed when using <literal>binary</literal> format. This option
+ is only supported for <command>COPY FROM</command> syntax.
+ If this option is omitted, any data type conversion errors will be raised immediately.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</refsect1>
@@ -564,6 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
amount to a considerable amount of wasted disk space if the failure
happened well into a large copy operation. You might wish to invoke
<command>VACUUM</command> to recover the wasted space.
+ To continue copying while skip conversion errors in a <command>COPY FROM</command>, you might wish to specify <literal>SAVE_ERROR</literal>.
</para>
<para>
@@ -572,6 +586,18 @@ COPY <replaceable class="parameter">count</replaceable>
null strings to null values and unquoted null strings to empty strings.
</para>
+ <para>
+ If the <literal>SAVE_ERROR</literal> option is specified and conversion errors occur while copying,
+ <productname>PostgreSQL</productname> will first check for the existence of the table <literal>COPY_ERRORS</literal>, then save the conversion error information to it.
+ If it does exist, but the table definition cannot use it to save the error, an error is raised, <command>COPY FROM</command> operation stops.
+ If it does not exist, <productname>PostgreSQL</productname> will try to create it before doing the actual copy operation.
+ The table <literal>COPY_ERRORS</literal> owner is the current <command>COPY FROM</command> operation's schema owner.
+ All the future errors related information generated while copying data to the same schema will automatically be saved to the same <literal>COPY_ERRORS</literal> table.
+ Currenly only the owner can read and write data to <literal>COPY_ERRORS</literal>.
+ Conversion errors include data type conversion failure, extra data or missing data in the source file.
+ <literal>COPY_ERRORS</literal> table detailed description listed in <xref linkend="copy-errors-table"/>.
+
+ </para>
</refsect1>
<refsect1>
@@ -588,7 +614,7 @@ COPY <replaceable class="parameter">count</replaceable>
output function, or acceptable to the input function, of each
attribute's data type. The specified null string is used in
place of columns that are null.
- <command>COPY FROM</command> will raise an error if any line of the
+ By default, if <literal>SAVE_ERROR</literal> not specified, <command>COPY FROM</command> will raise an error if any line of the
input file contains more or fewer columns than are expected.
</para>
@@ -962,6 +988,98 @@ versions of <productname>PostgreSQL</productname>.
check against somehow getting out of sync with the data.
</para>
</refsect3>
+
+ <refsect3>
+ <title> Table COPY_ERRORS </title>
+ <para>
+ If <literal>SAVE_ERROR</literal> specified, all the data type conversion errors while copying will automatically saved in <literal>COPY_ERRORS</literal>.
+ <xref linkend="copy-errors-table"/> shows <literal>COPY_ERRORS</literal> table's column name, data type, and description.
+ </para>
+
+ <table id="copy-errors-table">
+ <title>Error Saving table description </title>
+
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry>Column name</entry>
+ <entry>Data type</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry> <literal>userid</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The user generated the conversion error.
+ Refer <link linkend="catalog-pg-authid"><structname>pg_authid</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_authid</literal>. If the correspond <structfield>oid</structfield> deleted in <literal>pg_authid</literal>, this value become stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>copy_destination</literal> </entry>
+ <entry><type>oid</type></entry>
+ <entry>The <command>COPY FROM</command> operation destination table oid.
+ Refer <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>oid</structfield>.
+ There is no hard depenedency with <literal>pg_class</literal>. If the correspond <structfield>oid</structfield> deleted in <literal>pg_class</literal>, this value become stale.
+ </entry>
+ </row>
+
+ <row>
+ <entry> <literal>filename</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The path name of the <command>COPY FROM</command> input</entry>
+ </row>
+
+ <row>
+ <entry> <literal>lineno</literal> </entry>
+ <entry><type>bigint</type></entry>
+ <entry>Line number where the error occurred, counting from 1</entry>
+ </row>
+
+ <row>
+ <entry> <literal>line</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred line</entry>
+ </row>
+
+ <row>
+ <entry> <literal>colname</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Field where the error occurred</entry>
+ </row>
+
+ <row>
+ <entry> <literal>raw_field_value</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Raw content of the error occurred field</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_message </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error message</entry>
+ </row>
+
+ <row>
+ <entry> <literal>err_detail</literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>Detailed error message </entry>
+ </row>
+
+ <row>
+ <entry> <literal>errorcode </literal> </entry>
+ <entry><type>text</type></entry>
+ <entry>The error code </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+ </refsect3>
+
</refsect2>
</refsect1>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index cfad47b5..bc4af10a 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -419,6 +419,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -458,6 +459,13 @@ ProcessCopyOptions(ParseState *pstate,
freeze_specified = true;
opts_out->freeze = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "save_error") == 0)
+ {
+ if (save_error_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_specified = true;
+ opts_out->save_error = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "delimiter") == 0)
{
if (opts_out->delim)
@@ -598,6 +606,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index f4861652..a972ad87 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -29,7 +29,9 @@
#include "access/tableam.h"
#include "access/xact.h"
#include "access/xlog.h"
+#include "catalog/pg_authid.h"
#include "catalog/namespace.h"
+#include "catalog/pg_namespace.h"
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
@@ -38,6 +40,7 @@
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
#include "executor/tuptable.h"
+#include "executor/spi.h"
#include "foreign/fdwapi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
@@ -52,6 +55,7 @@
#include "utils/portal.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "utils/syscache.h"
/*
* No more than this many tuples per CopyMultiInsertBuffer
@@ -655,7 +659,8 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
-
+ if (cstate->opts.save_error)
+ Assert(cstate->escontext);
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +997,10 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ /* Soft error occured, skip this tuple. */
+ if (cstate->opts.save_error && cstate->line_error_occured)
+ continue;
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1297,6 +1306,20 @@ CopyFrom(CopyFromState cstate)
ExecResetTupleTable(estate->es_tupleTable, false);
+ if (cstate->opts.save_error)
+ {
+ Assert(cstate->copy_errors_nspname);
+
+ if (cstate->error_rows_cnt > 0)
+ {
+ ereport(NOTICE,
+ errmsg("%llu rows were skipped because of conversion error."
+ " Skipped rows saved to table %s.copy_errors",
+ (unsigned long long) cstate->error_rows_cnt,
+ cstate->copy_errors_nspname));
+ }
+ }
+
/* Allow the FDW to shut down */
if (target_resultRelInfo->ri_FdwRoutine != NULL &&
target_resultRelInfo->ri_FdwRoutine->EndForeignInsert != NULL)
@@ -1444,6 +1467,114 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR */
+ if (cstate->opts.save_error)
+ {
+ StringInfoData querybuf;
+ bool isnull;
+ bool copy_erros_table_ok;
+ Oid nsp_oid;
+ Oid save_userid;
+ Oid ownerId;
+ int save_sec_context;
+ const char *copy_errors_nspname;
+ HeapTuple tuple;
+
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->details_wanted = true;
+ cstate->escontext->error_occurred = false;
+
+ copy_errors_nspname = get_namespace_name(RelationGetNamespace(cstate->rel));
+ nsp_oid = get_namespace_oid(copy_errors_nspname, false);
+
+ initStringInfo(&querybuf);
+ /*
+ *
+ * Verify whether the nsp_oid.COPY_ERRORS table already exists, and if so,
+ * examine its column names and data types.
+ */
+ appendStringInfo(&querybuf,
+ "SELECT (array_agg(pa.attname ORDER BY pa.attnum) "
+ "= '{ctid,userid,copy_destination,filename,lineno, "
+ "line,colname,raw_field_value,err_message,err_detail,errorcode}') "
+ "AND (ARRAY_AGG(pt.typname ORDER BY pa.attnum) "
+ "= '{tid,oid,oid,text,int8,text,text,text,text,text,text}') "
+ "FROM pg_catalog.pg_attribute pa "
+ "JOIN pg_catalog.pg_class pc ON pc.oid = pa.attrelid "
+ "JOIN pg_catalog.pg_type pt ON pt.oid = pa.atttypid "
+ "JOIN pg_catalog.pg_namespace pn "
+ "ON pn.oid = pc.relnamespace WHERE ");
+ appendStringInfo(&querybuf,
+ "relname = $$copy_errors$$ AND pn.nspname = $$%s$$ "
+ " AND pa.attnum >= -1 AND NOT attisdropped ",
+ copy_errors_nspname);
+
+ if (SPI_connect() != SPI_OK_CONNECT)
+ elog(ERROR, "SPI_connect failed");
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_SELECT)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ copy_erros_table_ok = DatumGetBool(SPI_getbinval(SPI_tuptable->vals[0],
+ SPI_tuptable->tupdesc,
+ 1, &isnull));
+
+ tuple = SearchSysCache1(NAMESPACEOID, ObjectIdGetDatum(nsp_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("schema with OID %u does not exist", nsp_oid)));
+ ownerId = ((Form_pg_namespace) GETSTRUCT(tuple))->nspowner;
+ ReleaseSysCache(tuple);
+
+ cstate->copy_errors_owner = ownerId;
+
+ /*
+ * Switch to the schema owner's userid, so that the COPY_ERRORS table owned by
+ * that user.
+ */
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+
+ SetUserIdAndSecContext(ownerId,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ /* No copy_errors_nspname.COPY_ERRORS table then create it for holding all the potential error. */
+ if (isnull)
+ {
+ resetStringInfo(&querybuf);
+ appendStringInfo(&querybuf,
+ "CREATE TABLE %s.COPY_ERRORS( "
+ "USERID OID, COPY_DESTINATION OID, FILENAME TEXT,LINENO BIGINT "
+ ",LINE TEXT, COLNAME text, RAW_FIELD_VALUE TEXT "
+ ",ERR_MESSAGE TEXT, ERR_DETAIL TEXT, ERRORCODE TEXT)", copy_errors_nspname);
+
+ if (SPI_execute(querybuf.data, false, 0) != SPI_OK_UTILITY)
+ elog(ERROR, "SPI_exec failed: %s", querybuf.data);
+ }
+ else if(!copy_erros_table_ok)
+ ereport(ERROR,
+ (errmsg("table %s.COPY_ERRORS already exists. "
+ "cannot use it for COPY FROM error saving",
+ copy_errors_nspname)));
+
+ if (SPI_finish() != SPI_OK_FINISH)
+ elog(ERROR, "SPI_finish failed");
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->copy_errors_nspname = pstrdup(copy_errors_nspname);
+ }
+ else
+ {
+ cstate->copy_errors_nspname = NULL;
+ cstate->escontext = NULL;
+ cstate->copy_errors_owner = (Oid) 0;
+ }
+
+ cstate->error_rows_cnt = 0; /* set the default to 0 */
+ cstate->line_error_occured = false; /* default, assume conversion be ok. */
+
/* Convert convert_selectively name list to per-column flags */
if (cstate->opts.convert_selectively)
{
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index f5537345..ac204709 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -58,18 +58,21 @@
*/
#include "postgres.h"
+#include "access/heapam.h"
#include <ctype.h>
#include <unistd.h>
#include <sys/stat.h>
-
+#include <catalog/namespace.h>
#include "commands/copy.h"
#include "commands/copyfrom_internal.h"
#include "commands/progress.h"
#include "executor/executor.h"
+#include "executor/spi.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -880,16 +883,85 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
int fldct;
int fieldno;
char *string;
+ char *errmsg_extra;
+ Oid save_userid = InvalidOid;
+ int save_sec_context = -1;
+ HeapTuple copy_errors_tup = NULL;
+ Relation copy_errorsrel = NULL;
+ TupleDesc copy_errors_tupDesc = NULL;
+ Datum t_values[10] = {0};
+ bool t_isnull[10] = {0};
/* read raw fields in the next line */
if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
return false;
+ if (cstate->opts.save_error)
+ {
+ /*
+ * Open the copy_errors relation. we also need current userid for the later heap inserts.
+ *
+ */
+ copy_errorsrel = table_open(RelnameGetRelid("copy_errors"), RowExclusiveLock);
+ copy_errors_tupDesc = copy_errorsrel->rd_att;
+ GetUserIdAndSecContext(&save_userid, &save_sec_context);
+ }
+
+ /* reset line_error_occured to false for next new line. */
+ if (cstate->line_error_occured)
+ cstate->line_error_occured = false;
+
/* check for overflowing fields */
if (attr_count > 0 && fldct > attr_count)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
+ {
+ if(cstate->opts.save_error)
+ {
+ errmsg_extra = pstrdup("extra data after last expected column");
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(
+ cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(errmsg_extra);
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("extra data after last expected column")));
+ }
fieldno = 0;
@@ -901,10 +973,55 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
Form_pg_attribute att = TupleDescAttr(tupDesc, m);
if (fieldno >= fldct)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("missing data for column \"%s\"",
- NameStr(att->attname))));
+ {
+ if(cstate->opts.save_error)
+ {
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = (Datum) 0;
+ t_isnull[5] = true;
+ t_values[6] = (Datum) 0;
+ t_isnull[6] = true;
+ t_values[7] = CStringGetTextDatum(
+ psprintf("missing data for column \"%s\"", NameStr(att->attname)));
+ t_isnull[7] = false;
+ t_values[8] = (Datum) 0;
+ t_isnull[8] = true;
+ t_values[9] = CStringGetTextDatum(
+ unpack_sql_state(ERRCODE_BAD_COPY_FILE_FORMAT));
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+ cstate->line_error_occured = true;
+ cstate->error_rows_cnt++;
+ table_close(copy_errorsrel, RowExclusiveLock);
+ return true;
+ }
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(att->attname))));
+ }
+
string = field_strings[fieldno++];
if (cstate->convert_select_flags &&
@@ -956,15 +1073,91 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ {
+ /*
+ *
+ * InputFunctionCall is more faster than InputFunctionCallSafe.
+ *
+ */
+ if(!cstate->opts.save_error)
+ values[m] = InputFunctionCall(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod);
+ else
+ {
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ char *err_detail;
+ char *err_code;
+ err_code = pstrdup(unpack_sql_state(cstate->escontext->error_data->sqlerrcode));
+ if (!cstate->escontext->error_data->detail)
+ err_detail = NULL;
+ else
+ err_detail = cstate->escontext->error_data->detail;
+
+ t_values[0] = ObjectIdGetDatum(save_userid);
+ t_isnull[0] = false;
+ t_values[1] = ObjectIdGetDatum(cstate->rel->rd_rel->oid);
+ t_isnull[1] = false;
+ t_values[2] = CStringGetTextDatum(cstate->filename ? cstate->filename : "STDIN");
+ t_isnull[2] = false;
+ t_values[3] = Int64GetDatum((long long) cstate->cur_lineno);
+ t_isnull[3] = false;
+ t_values[4] = CStringGetTextDatum(cstate->line_buf.data);
+ t_isnull[4] = false;
+ t_values[5] = CStringGetTextDatum(cstate->cur_attname);
+ t_isnull[5] = false;
+ t_values[6] = CStringGetTextDatum(string);
+ t_isnull[6] = false;
+ t_values[7] = CStringGetTextDatum(cstate->escontext->error_data->message);
+ t_isnull[7] = false;
+ t_values[8] = err_detail ? CStringGetTextDatum(err_detail) : (Datum) 0;
+ t_isnull[8] = err_detail ? false: true;
+ t_values[9] = CStringGetTextDatum(err_code);
+ t_isnull[9] = false;
+
+ copy_errors_tup = heap_form_tuple(copy_errors_tupDesc,
+ t_values,
+ t_isnull);
+ /* using copy_errors owner do the simple_heap_insert */
+ SetUserIdAndSecContext(cstate->copy_errors_owner,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE |
+ SECURITY_NOFORCE_RLS);
+
+ simple_heap_insert(copy_errorsrel, copy_errors_tup);
+
+ /* Restore userid and security context */
+ SetUserIdAndSecContext(save_userid, save_sec_context);
+
+ /* line error occured, set it once per line */
+ if (!cstate->line_error_occured)
+ cstate->line_error_occured = true;
+ /* reset ErrorSaveContext */
+ cstate->escontext->error_occurred = false;
+ cstate->escontext->details_wanted = true;
+ memset(cstate->escontext->error_data,0, sizeof(ErrorData));
+ }
+ }
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
}
+ /* record error rows count. */
+ if (cstate->line_error_occured)
+ {
+ cstate->error_rows_cnt++;
+ Assert(cstate->opts.save_error);
+ }
+ if (cstate->opts.save_error)
+ table_close(copy_errorsrel, RowExclusiveLock);
Assert(fieldno == attr_count);
}
else
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4b175ef6..fc69420e 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -778,7 +778,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
SEQUENCE SEQUENCES
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
@@ -3473,6 +3473,10 @@ copy_opt_item:
{
$$ = makeDefElem("encoding", (Node *) makeString($2), @1);
}
+ | SAVE_ERROR
+ {
+ $$ = makeDefElem("save_error", (Node *) makeBoolean(true), @1);
+ }
;
/* The following exist for backward compatibility with very old versions */
@@ -17768,6 +17772,7 @@ unreserved_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
@@ -18395,6 +18400,7 @@ bare_label_keyword:
| ROWS
| RULE
| SAVEPOINT
+ | SAVE_ERROR
| SCALAR
| SCHEMA
| SCHEMAS
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 04980118..e6a358e0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2890,7 +2890,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index f2cca0b9..aa560dbb 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -43,6 +43,7 @@ typedef struct CopyFormatOptions
bool binary; /* binary format? */
bool freeze; /* freeze rows on loading? */
bool csv_mode; /* Comma Separated Value format? */
+ bool save_error; /* save error to a table? */
CopyHeaderChoice header_line; /* header line? */
char *null_print; /* NULL marker string (server encoding!) */
int null_print_len; /* length of same */
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 5ec41589..2c3b7b42 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,11 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ Oid copy_errors_owner; /* the owner of copy_errors table */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 error_rows_cnt; /* total number of rows that have errors */
+ const char *copy_errors_nspname; /* the copy_errors's namespace */
+ bool line_error_occured; /* does this line conversion error happened */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f88a6c9a..b6f7ed48 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -390,6 +390,7 @@ PG_KEYWORD("routines", ROUTINES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("row", ROW, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("rows", ROWS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("rule", RULE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("save_error", SAVE_ERROR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("savepoint", SAVEPOINT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("scalar", SCALAR, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("schema", SCHEMA, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c..a2c6bf5aa 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -564,6 +564,118 @@ ERROR: conflicting or redundant options
LINE 1: ... b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL...
^
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+ERROR: cannot specify SAVE_ERROR in BINARY mode
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+ERROR: conflicting or redundant options
+LINE 1: COPY save_error_csv FROM STDIN WITH (save_error, save_error ...
+ ^
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+ERROR: table public.COPY_ERRORS already exists. cannot use it for COPY FROM error saving
+drop table COPY_ERRORS;
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+ a | b | c | b_null | b_empty
+---+---+------+--------+---------
+ 2 | | NULL | f | t
+(1 row)
+
+DROP TABLE save_error_csv;
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+NOTICE: 10 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+ relname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+---------------+----------+--------+--------------------------------------------+---------+-------------------------+-----------------------------------------------------------------+---------------------------+-----------
+ check_ign_err | STDIN | 1 | 1 {1} 1 1 extra | NULL | NULL | extra data after last expected column | NULL | 22P04
+ check_ign_err | STDIN | 2 | 2 | NULL | NULL | missing data for column "m" | NULL | 22P04
+ check_ign_err | STDIN | 3 | \n {1} 1 \- | n | +| invalid input syntax for type integer: " +| NULL | 22P02
+ | | | | | | " | |
+ check_ign_err | STDIN | 4 | a {2} 2 \r | n | a | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 5 | 3 {\3} 3333333333 \n | m | {\x03} | invalid input syntax for type integer: "\x03" | NULL | 22P02
+ check_ign_err | STDIN | 6 | 0x11 {3,} 3333333333 \\. | m | {3,} | malformed array literal: "{3,}" | Unexpected "}" character. | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | n | d | invalid input syntax for type integer: "d" | NULL | 22P02
+ check_ign_err | STDIN | 7 | d {3,1/} 3333333333 \\0 | m | {3,1/} | invalid input syntax for type integer: "1/" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | n | e | invalid input syntax for type integer: "e" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | m | {3,\x01} | invalid input syntax for type integer: "\x01" | NULL | 22P02
+ check_ign_err | STDIN | 8 | e {3,\1} -3323879289873933333333 \n | k | -3323879289873933333333 | value "-3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | n | f | invalid input syntax for type integer: "f" | NULL | 22P02
+ check_ign_err | STDIN | 9 | f {3,1} 3323879289873933333333 \r | k | 3323879289873933333333 | value "3323879289873933333333" is out of range for type bigint | NULL | 22003
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | n | b | invalid input syntax for type integer: "b" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | m | {a, 4} | invalid input syntax for type integer: "a" | NULL | 22P02
+ check_ign_err | STDIN | 10 | b {a, 4} 1.1 h | k | 1.1 | invalid input syntax for type bigint: "1.1" | NULL | 22P02
+(16 rows)
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK to s1;
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+NOTICE: 2 rows were skipped because of conversion error. Skipped rows saved to table copy_errors_test.copy_errors
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+ relname | rolname | filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+-----------------+----------------+----------+--------+----------------------------+---------+-----------------+-------------------------------------------------------------------+------------------------------------------+-----------
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | b | -[a\,z) | malformed range literal: "-[a\,z)" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 1 | ,-[a\","z),[a","-inf) | c | [a,-inf) | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | a | (,a),( | malformed range literal: "(,a),(" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | b | ,a),() | malformed range literal: ",a),()" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user13 | STDIN | 2 | (",a),(",",a),()",a); | c | a); | malformed range literal: "a);" | Missing left parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | a | (a,)) | malformed range literal: "(a,))" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | b | (],a) | malformed range literal: "(],a)" | Missing comma after lower bound. | 22P02
+ textrange_input | regress_user12 | STDIN | 1 | (a",")),(]","a),(a","]) | c | (a,]) | malformed range literal: "(a,])" | Junk after right parenthesis or bracket. | 22P02
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | a | [z,a] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | b | [z,2] | range lower bound must be less than or equal to range upper bound | NULL | 22000
+ textrange_input | regress_user12 | STDIN | 2 | [z","a],[z","2],[(","",")] | c | [(,",)] | malformed range literal: "[(,",)]" | Unexpected end of input. | 22P02
+(11 rows)
+
+--owner allowed to drop the table.
+drop table copy_errors;
+--should fail. no priviledge
+select * from public.copy_errors;
+ERROR: permission denied for table copy_errors
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
create table check_con_tbl (f1 int);
@@ -822,3 +934,28 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
ERROR: COPY DEFAULT only available using COPY FROM
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+NOTICE: 3 rows were skipped because of conversion error. Skipped rows saved to table public.copy_errors
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+ filename | lineno | line | colname | raw_field_value | err_message | err_detail | errorcode
+----------+--------+----------------------------------+----------+------------------+-------------------------------------------------------------+------------+-----------
+ STDIN | 1 | k value '2022-07-04' | id | k | invalid input syntax for type integer: "k" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | id | z | invalid input syntax for type integer: "z" | | 22P02
+ STDIN | 2 | z \D '2022-07-03ASKL' | ts_value | '2022-07-03ASKL' | invalid input syntax for type timestamp: "'2022-07-03ASKL'" | | 22007
+ STDIN | 3 | s \D \D | id | s | invalid input syntax for type integer: "s" | | 22P02
+(4 rows)
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60..a37986df 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -374,6 +374,106 @@ BEGIN;
COPY forcetest (a, b, c) FROM STDIN WITH (FORMAT csv, FORCE_NULL *, FORCE_NULL(b));
ROLLBACK;
+--
+-- tests for SAVE_ERROR option with force_not_null, force_null
+\pset null NULL
+CREATE TABLE save_error_csv(
+ a INT NOT NULL,
+ b TEXT NOT NULL,
+ c TEXT
+);
+
+--save_error not allowed in binary mode
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT binary);
+
+-- redundant options not allowed.
+COPY save_error_csv FROM STDIN WITH (save_error, save_error off);
+
+create table COPY_ERRORS();
+--should fail. since table COPY_ERRORS already exists.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error);
+
+drop table COPY_ERRORS;
+
+--with FORCE_NOT_NULL and FORCE_NULL.
+COPY save_error_csv (a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NOT_NULL(b), FORCE_NULL(c));
+z,,""
+\0,,
+2,,
+\.
+
+SELECT *, b is null as b_null, b = '' as b_empty FROM save_error_csv;
+DROP TABLE save_error_csv;
+
+-- save error with extra data and missing data some column.
+---normal data type conversion error case.
+CREATE TABLE check_ign_err (n int, m int[], k bigint, l text);
+COPY check_ign_err FROM STDIN WITH (save_error);
+1 {1} 1 1 extra
+2
+\n {1} 1 \-
+a {2} 2 \r
+3 {\3} 3333333333 \n
+0x11 {3,} 3333333333 \\.
+d {3,1/} 3333333333 \\0
+e {3,\1} -3323879289873933333333 \n
+f {3,1} 3323879289873933333333 \r
+b {a, 4} 1.1 h
+5 {5} 5 \\
+\.
+
+select pc.relname, ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+from copy_errors ce join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'check_ign_err';
+
+DROP TABLE check_ign_err;
+truncate COPY_ERRORS;
+
+--(type textrange was already made in test_setup.sql)
+--using textrange doing test
+begin;
+CREATE USER regress_user12;
+CREATE USER regress_user13;
+CREATE SCHEMA IF NOT EXISTS copy_errors_test AUTHORIZATION regress_user12;
+SET LOCAL search_path TO copy_errors_test;
+
+GRANT USAGE on schema copy_errors_test to regress_user12,regress_user13;
+GRANT CREATE on schema copy_errors_test to regress_user12;
+set role regress_user12;
+CREATE TABLE textrange_input(a public.textrange, b public.textrange, c public.textrange);
+GRANT insert on textrange_input to regress_user13;
+
+set role regress_user13;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+,-[a\","z),[a","-inf)
+(",a),(",",a),()",a);
+\.
+
+SAVEPOINT s1;
+--should fail. no priviledge
+select * from copy_errors_test.copy_errors;
+
+ROLLBACK to s1;
+
+set role regress_user12;
+COPY textrange_input(a, b, c) FROM STDIN WITH (save_error,FORMAT csv, FORCE_NULL *);
+(a",")),(]","a),(a","])
+[z","a],[z","2],[(","",")]
+\.
+
+SELECT pc.relname,pr.rolname,ce.filename,ce.lineno,ce.line,ce.colname,
+ ce.raw_field_value,ce.err_message,ce.err_detail,ce.errorcode
+FROM copy_errors_test.copy_errors ce
+JOIN pg_class pc ON pc.oid = ce.copy_destination
+JOIN pg_roles pr ON pr.oid = ce.userid;
+
+--owner allowed to drop the table.
+drop table copy_errors;
+
+--should fail. no priviledge
+select * from public.copy_errors;
+ROLLBACK;
\pset null ''
-- test case with whole-row Var in a check constraint
@@ -609,3 +709,26 @@ truncate copy_default;
-- DEFAULT cannot be used in COPY TO
copy (select 1 as test) TO stdout with (default '\D');
+
+-- DEFAULT WITH SAVE_ERROR.
+create table copy_default_error_save (
+ id integer,
+ text_value text not null default 'test',
+ ts_value timestamp without time zone not null default '2022-07-05'
+);
+copy copy_default_error_save from stdin with (save_error, default '\D');
+k value '2022-07-04'
+z \D '2022-07-03ASKL'
+s \D \D
+\.
+
+select ce.filename,ce.lineno,ce.line,
+ ce.colname, ce.raw_field_value,
+ ce.err_message, ce.err_detail,ce.errorcode
+from public.copy_errors ce
+join pg_class pc on pc.oid = ce.copy_destination
+where pc.relname = 'copy_default_error_save'
+order by lineno, colname;
+
+drop table copy_default_error_save, copy_errors;
+truncate copy_default;
\ No newline at end of file
--
2.34.1
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-09 14:36 ` torikoshia <[email protected]>
2024-01-10 07:42 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: torikoshia @ 2024-01-09 14:36 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Tue, Dec 19, 2023 at 10:14 AM Masahiko Sawada <[email protected]>
wrote:
> If we want only such a feature we need to implement it together (the
> patch could be split, though). But if some parts of the feature are
> useful for users as well, I'd recommend implementing it incrementally.
> That way, the patches can get small and it would be easy for reviewers
> and committers to review/commit them.
Jian, how do you think this comment?
Looking back at the discussion so far, it seems that not everyone thinks
saving table information is the best idea[1] and some people think just
skipping error data is useful.[2]
Since there are issues to be considered from the design such as
physical/logical replication treatment, putting error information to
table is likely to take time for consensus building and development.
Wouldn't it be better to follow the following advice and develop the
functionality incrementally?
On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada
<sawada(dot)mshk(at)gmail(dot)com> wrote:
> So I'm thinking we may be able to implement this
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
> be to support logging destinations such as server logs and tables.
Attached a patch for this "first step" with reference to v7 patch, which
logged errors and simpler than latest one.
- This patch adds new option SAVE_ERROR_TO, but currently only supports
'none', which means just skips error data. It is expected to support
'log' and 'table'.
- This patch Skips just soft errors and don't handle other errors such
as missing column data.
BTW I have question and comment about v15 patch:
> + {
> + /*
> + *
> + * InputFunctionCall is more faster than
> InputFunctionCallSafe.
> + *
> + */
Have you measured this?
When I tested it in an older patch, there were no big difference[3].
> - SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY
SELECT
> + SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P
SECURITY SELECT
There was a comment that we shouldn't add new keyword for this[4].
I left as it was in v7 patch regarding these points.
[1]
https://www.postgresql.org/message-id/20231109002600.fuihn34bjqqgmbjm%40awork3.anarazel.de
[2]
https://www.postgresql.org/message-id/CAD21AoCeEOBN49fu43e6tBTynnswugA3oZ5AZvLeyDCpxpCXPg%40mail.gma...
[3]
https://www.postgresql.org/message-id/19551e8c2717c24689913083f841ddb5%40oss.nttdata.com
[4]
https://www.postgresql.org/message-id/20230322175000.qbdctk7bnmifh5an%40awork3.anarazel.de
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
Attachments:
[text/x-diff] v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch (14.5K, ../../[email protected]/2-v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch)
download | inline diff:
From 675b8b8408e23f22940a99b40cb7ec3e1b36cac3 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <[email protected]>
Date: Tue, 9 Jan 2024 23:10:14 +0900
Subject: [PATCH v1] Add new COPY option SAVE_ERROR_TO
Currently when source data contains unexpected data regarding data type or
range, entire COPY fails. However, in some cases such data can be ignored and
just copying normal data is preferable.
This patch adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying data.
Currently SAVE_ERROR_TO only supports 'none'. This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, referenced with jian he's patch.
---
doc/src/sgml/ref/copy.sgml | 20 +++++++++++++-
src/backend/commands/copy.c | 19 ++++++++++++++
src/backend/commands/copyfrom.c | 33 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 16 +++++++++---
src/bin/psql/tab-complete.c | 7 ++++-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 3 +++
src/test/regress/expected/copy2.out | 28 ++++++++++++++++++++
src/test/regress/sql/copy2.sql | 27 +++++++++++++++++++
9 files changed, 148 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c33..87f2b3e7a2 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,22 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies where to save error information when there are malformed data in
+ the input. If this option is specified, <command>COPY</command> skips
+ malformed data and continues copying data.
+ Currently only <literal>none</literal> is supported.
+ This option is allowed only in <command>COPY FROM</command>, and only when
+ not using <literal>binary</literal> format.
+ Note that this is only supported in current <command>COPY</command>
+ syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +573,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d7..5e5e8a5f34 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -571,6 +571,20 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ char *location = defGetString(defel);
+
+ if (opts_out->save_error_to)
+ errorConflictingDefElem(defel, pstate);
+ else if (strcmp(location, "none") == 0)
+ opts_out->save_error_to = location;
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", location),
+ parser_errposition(pstate, defel->location)));
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +612,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c..d909123cd1 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -752,6 +753,14 @@ CopyFrom(CopyFromState cstate)
ti_options |= TABLE_INSERT_FROZEN;
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to)
+ {
+ ErrorSaveContext escontext = {T_ErrorSaveContext};
+ escontext.details_wanted = true;
+ cstate->escontext = escontext;
+ }
+
/*
* We need a ResultRelInfo so we can use the regular executor's
* index-entry-making machinery. (There used to be a huge amount of code
@@ -992,6 +1001,25 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (cstate->escontext.error_occurred)
+ {
+ ErrorSaveContext new_escontext = {T_ErrorSaveContext};
+
+ /* Currently only "none" is supported */
+ Assert(strcmp(cstate->opts.save_error_to, "none") == 0);
+
+ ExecClearTuple(myslot);
+
+ new_escontext.details_wanted = true;
+ cstate->escontext = new_escontext;
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1281,6 +1309,11 @@ CopyFrom(CopyFromState cstate)
CopyMultiInsertInfoFlush(&multiInsertInfo, NULL, &processed);
}
+ if (cstate->opts.save_error_to && cstate->num_errors > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors));
+
/* Done, clean up */
error_context_stack = errcallback.previous;
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f645..0dd49d85e6 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -956,10 +957,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) &cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e4..efe2b7cc10 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2f..f890b66f26 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -62,6 +62,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ char *save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a907..e2a8f9dd6e 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,8 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext escontext; /* soft error trapper during in_functions execution */
+ uint64 num_errors; /* total number of rows which contained soft errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07..4a1777a4fa 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -82,6 +82,8 @@ COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x to stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +96,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY save_error_to "unsupported" not recognized
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +716,26 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+WARNING: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +750,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f6086..17c5764b42 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -70,12 +70,14 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x to stdin (format BINARY, save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +496,29 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +533,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
base-commit: d596736a499858de800cabb241c0107c978f1b95
--
2.39.2
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-10 07:42 ` Masahiko Sawada <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Masahiko Sawada @ 2024-01-10 07:42 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: jian he <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Tue, Jan 9, 2024 at 11:36 PM torikoshia <[email protected]> wrote:
>
> On Tue, Dec 19, 2023 at 10:14 AM Masahiko Sawada <[email protected]>
> wrote:
> > If we want only such a feature we need to implement it together (the
> > patch could be split, though). But if some parts of the feature are
> > useful for users as well, I'd recommend implementing it incrementally.
> > That way, the patches can get small and it would be easy for reviewers
> > and committers to review/commit them.
>
> Jian, how do you think this comment?
>
> Looking back at the discussion so far, it seems that not everyone thinks
> saving table information is the best idea[1] and some people think just
> skipping error data is useful.[2]
>
> Since there are issues to be considered from the design such as
> physical/logical replication treatment, putting error information to
> table is likely to take time for consensus building and development.
>
> Wouldn't it be better to follow the following advice and develop the
> functionality incrementally?
Yeah, I'm still thinking it's better to implement this feature
incrementally. Given we're closing to feature freeze, I think it's
unlikely to get the whole feature into PG17 since there are still many
design discussions we need in addition to what Torikoshi-san pointed
out. The feature like "ignore errors" or "logging errors" would have
higher possibilities. Even if we get only these parts of the whole
"error table" feature into PG17, it will make it much easier to
implement "error tables" feature.
>
> On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada
> <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > So I'm thinking we may be able to implement this
> > feature incrementally. The first step would be something like an
> > option to ignore all errors or an option to specify the maximum number
> > of errors to tolerate before raising an ERROR. The second step would
> > be to support logging destinations such as server logs and tables.
>
>
> Attached a patch for this "first step" with reference to v7 patch, which
> logged errors and simpler than latest one.
> - This patch adds new option SAVE_ERROR_TO, but currently only supports
> 'none', which means just skips error data. It is expected to support
> 'log' and 'table'.
> - This patch Skips just soft errors and don't handle other errors such
> as missing column data.
Seems promising. I'll look at the patch.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-11 03:13 ` jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-11 03:13 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: vignesh C <[email protected]>; Masahiko Sawada <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Tue, Jan 9, 2024 at 10:36 PM torikoshia <[email protected]> wrote:
>
> On Tue, Dec 19, 2023 at 10:14 AM Masahiko Sawada <[email protected]>
> wrote:
> > If we want only such a feature we need to implement it together (the
> > patch could be split, though). But if some parts of the feature are
> > useful for users as well, I'd recommend implementing it incrementally.
> > That way, the patches can get small and it would be easy for reviewers
> > and committers to review/commit them.
>
> Jian, how do you think this comment?
>
> Looking back at the discussion so far, it seems that not everyone thinks
> saving table information is the best idea[1] and some people think just
> skipping error data is useful.[2]
>
> Since there are issues to be considered from the design such as
> physical/logical replication treatment, putting error information to
> table is likely to take time for consensus building and development.
>
> Wouldn't it be better to follow the following advice and develop the
> functionality incrementally?
>
> On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada
> <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > So I'm thinking we may be able to implement this
> > feature incrementally. The first step would be something like an
> > option to ignore all errors or an option to specify the maximum number
> > of errors to tolerate before raising an ERROR. The second step would
> > be to support logging destinations such as server logs and tables.
>
>
> Attached a patch for this "first step" with reference to v7 patch, which
> logged errors and simpler than latest one.
> - This patch adds new option SAVE_ERROR_TO, but currently only supports
> 'none', which means just skips error data. It is expected to support
> 'log' and 'table'.
> - This patch Skips just soft errors and don't handle other errors such
> as missing column data.
Hi.
I made the following change based on your patch
(v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch)
* when specified SAVE_ERROR_TO, move the initialization of
ErrorSaveContext to the function BeginCopyFrom.
I think that's the right place to initialize struct CopyFromState field.
* I think your patch when there are N rows have malformed data, then it
will initialize N ErrorSaveContext.
In the struct CopyFromStateData, I changed it to ErrorSaveContext *escontext.
So if an error occurred, you can just set the escontext accordingly.
* doc: mention "If this option is omitted, <command>COPY</command>
stops operation at the first error."
* Since we only support 'none' for now, 'none' means we don't want
ErrorSaveContext metadata,
so we should set cstate->escontext->details_wanted to false.
> BTW I have question and comment about v15 patch:
>
> > + {
> > + /*
> > + *
> > + * InputFunctionCall is more faster than
> > InputFunctionCallSafe.
> > + *
> > + */
>
> Have you measured this?
> When I tested it in an older patch, there were no big difference[3].
Thanks for pointing it out, I probably was over thinking.
> > - SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY
> SELECT
> > + SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P
> SECURITY SELECT
>
> There was a comment that we shouldn't add new keyword for this[4].
>
Thanks for pointing it out.
Attachments:
[application/octet-stream] v1-0001-minor-refactor.no-cfbot (5.5K, ../../CACJufxEqr5kukVazRQFV_XNww53SS67rJn0DUZ1d9iBKp6=yOg@mail.gmail.com/2-v1-0001-minor-refactor.no-cfbot)
download
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-12 02:58 ` torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-01-12 02:58 UTC (permalink / raw)
To: jian he <[email protected]>; [email protected]; +Cc: vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Wed, Jan 10, 2024 at 4:42 PM Masahiko Sawada <[email protected]>
wrote:
> Yeah, I'm still thinking it's better to implement this feature
> incrementally. Given we're closing to feature freeze, I think it's
> unlikely to get the whole feature into PG17 since there are still many
> design discussions we need in addition to what Torikoshi-san pointed
> out. The feature like "ignore errors" or "logging errors" would have
> higher possibilities. Even if we get only these parts of the whole
> "error table" feature into PG17, it will make it much easier to
implement "error tables" feature.
+1.
I'm also going to make patch for "logging errors", since this
functionality is isolated from v7 patch.
> Seems promising. I'll look at the patch.
Thanks a lot!
Sorry to attach v2 if you already reviewed v1..
On 2024-01-11 12:13, jian he wrote:
> On Tue, Jan 9, 2024 at 10:36 PM torikoshia <[email protected]>
> wrote:
>>
>> On Tue, Dec 19, 2023 at 10:14 AM Masahiko Sawada
>> <[email protected]>
>> wrote:
>> > If we want only such a feature we need to implement it together (the
>> > patch could be split, though). But if some parts of the feature are
>> > useful for users as well, I'd recommend implementing it incrementally.
>> > That way, the patches can get small and it would be easy for reviewers
>> > and committers to review/commit them.
>>
>> Jian, how do you think this comment?
>>
>> Looking back at the discussion so far, it seems that not everyone
>> thinks
>> saving table information is the best idea[1] and some people think
>> just
>> skipping error data is useful.[2]
>>
>> Since there are issues to be considered from the design such as
>> physical/logical replication treatment, putting error information to
>> table is likely to take time for consensus building and development.
>>
>> Wouldn't it be better to follow the following advice and develop the
>> functionality incrementally?
>>
>> On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada
>> <sawada(dot)mshk(at)gmail(dot)com> wrote:
>> > So I'm thinking we may be able to implement this
>> > feature incrementally. The first step would be something like an
>> > option to ignore all errors or an option to specify the maximum number
>> > of errors to tolerate before raising an ERROR. The second step would
>> > be to support logging destinations such as server logs and tables.
>>
>>
>> Attached a patch for this "first step" with reference to v7 patch,
>> which
>> logged errors and simpler than latest one.
>> - This patch adds new option SAVE_ERROR_TO, but currently only
>> supports
>> 'none', which means just skips error data. It is expected to support
>> 'log' and 'table'.
>> - This patch Skips just soft errors and don't handle other errors such
>> as missing column data.
>
> Hi.
> I made the following change based on your patch
> (v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch)
>
> * when specified SAVE_ERROR_TO, move the initialization of
> ErrorSaveContext to the function BeginCopyFrom.
> I think that's the right place to initialize struct CopyFromState
> field.
> * I think your patch when there are N rows have malformed data, then it
> will initialize N ErrorSaveContext.
> In the struct CopyFromStateData, I changed it to ErrorSaveContext
> *escontext.
> So if an error occurred, you can just set the escontext accordingly.
> * doc: mention "If this option is omitted, <command>COPY</command>
> stops operation at the first error."
> * Since we only support 'none' for now, 'none' means we don't want
> ErrorSaveContext metadata,
> so we should set cstate->escontext->details_wanted to false.
>
>> BTW I have question and comment about v15 patch:
>>
>> > + {
>> > + /*
>> > + *
>> > + * InputFunctionCall is more faster than
>> > InputFunctionCallSafe.
>> > + *
>> > + */
>>
>> Have you measured this?
>> When I tested it in an older patch, there were no big difference[3].
> Thanks for pointing it out, I probably was over thinking.
>
>> > - SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P
>> SECURITY
>> SELECT
>> > + SAVEPOINT SAVE_ERROR SCALAR SCHEMA SCHEMAS SCROLL SEARCH
>> SECOND_P
>> SECURITY SELECT
>>
>> There was a comment that we shouldn't add new keyword for this[4].
>>
> Thanks for pointing it out.
Thanks for reviewing!
Updated the patch merging your suggestions except below points:
> + cstate->num_errors = 0;
Since cstate is already initialized in below lines, this may be
redundant.
| /* Allocate workspace and zero all fields */
| cstate = (CopyFromStateData *) palloc0(sizeof(CopyFromStateData));
> + Assert(!cstate->escontext->details_wanted);
I'm not sure this is necessary, considering we're going to add other
options like 'table' and 'log', which need details_wanted soon.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
Attachments:
[text/x-diff] v2-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch (15.3K, ../../[email protected]/2-v2-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch)
download | inline diff:
From a3f14a0e7e9a7b5fb961ad6b6b7b163cf6534a26 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <[email protected]>
Date: Fri, 12 Jan 2024 11:32:00 +0900
Subject: [PATCH v2] Add new COPY option SAVE_ERROR_TO
Currently when source data contains unexpected data regarding data type or
range, entire COPY fails. However, in some cases such data can be ignored and
just copying normal data is preferable.
This patch adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.
Currently SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, referenced with jian he's patch.
---
doc/src/sgml/ref/copy.sgml | 21 +++++++++++-
src/backend/commands/copy.c | 19 +++++++++++
src/backend/commands/copyfrom.c | 43 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 17 +++++++---
src/bin/psql/tab-complete.c | 7 +++-
src/include/commands/copy.h | 1 +
src/include/commands/copyfrom_internal.h | 3 ++
src/test/regress/expected/copy2.out | 28 +++++++++++++++
src/test/regress/sql/copy2.sql | 27 +++++++++++++++
9 files changed, 159 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 18ecc69c33..71941c4ee5 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,23 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies save error information to <replaceable class="parameter">
+ location</replaceable> when there are malformed data in the input.
+ If this option is specified, <command>COPY</command> skips malformed data
+ and continues copying data.
+ Currently only <literal>none</literal> is supported.
+ If this option is omitted, <command>COPY</command> stops operation at the
+ first error.
+ This option is allowed only in <command>COPY FROM</command>, and only when
+ not using <literal>binary</literal> format.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +574,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d7..5e5e8a5f34 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -571,6 +571,20 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ char *location = defGetString(defel);
+
+ if (opts_out->save_error_to)
+ errorConflictingDefElem(defel, pstate);
+ else if (strcmp(location, "none") == 0)
+ opts_out->save_error_to = location;
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", location),
+ parser_errposition(pstate, defel->location)));
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +612,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c..48484a2597 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -656,6 +657,9 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
+ if (cstate->opts.save_error_to)
+ Assert(cstate->escontext);
+
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +996,26 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ if (cstate->opts.save_error_to && cstate->escontext->error_occurred)
+ {
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (strcmp(cstate->opts.save_error_to, "none") == 0)
+ /*
+ * Just make ErrorSaveContext ready for the next NextCopyFrom.
+ * Since we don't set details_wanted and error_data is not to be
+ * filled, just resetting error_occurred is enough.
+ */
+ cstate->escontext->error_occurred = false;
+ else
+ elog(ERROR, "unexpected SAVE_ERROR_TO location : %s",
+ cstate->opts.save_error_to);
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1281,6 +1305,11 @@ CopyFrom(CopyFromState cstate)
CopyMultiInsertInfoFlush(&multiInsertInfo, NULL, &processed);
}
+ if (cstate->opts.save_error_to && cstate->num_errors > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors));
+
/* Done, clean up */
error_context_stack = errcallback.previous;
@@ -1419,6 +1448,20 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to)
+ {
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->error_occurred = false;
+
+ /* Currently we only support "none". We'll add other options later */
+ if (strcmp(cstate->opts.save_error_to, "none") == 0)
+ cstate->escontext->details_wanted = false;
+ }
+ else
+ cstate->escontext = NULL;
+
/* Convert FORCE_NULL name list to per-column flags, check validity */
cstate->opts.force_null_flags = (bool *) palloc0(num_phys_attrs * sizeof(bool));
if (cstate->opts.force_null_all)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f645..7041815dee 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -955,11 +956,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ else if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e4..efe2b7cc10 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2f..f890b66f26 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -62,6 +62,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ char *save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a907..3744fac017 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,8 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions execution */
+ uint64 num_errors; /* total number of rows which contained soft errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07..4a1777a4fa 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -82,6 +82,8 @@ COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x to stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +96,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY save_error_to "unsupported" not recognized
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +716,26 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+WARNING: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +750,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f6086..17c5764b42 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -70,12 +70,14 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x to stdin (format BINARY, save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +496,29 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +533,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
base-commit: 08c3ad27eb5348d0cbffa843a3edb11534f9904a
--
2.39.2
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-13 14:19 ` jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-13 14:19 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: [email protected]; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Fri, Jan 12, 2024 at 10:59 AM torikoshia <[email protected]> wrote:
>
>
> Thanks for reviewing!
>
> Updated the patch merging your suggestions except below points:
>
> > + cstate->num_errors = 0;
>
> Since cstate is already initialized in below lines, this may be
> redundant.
>
> | /* Allocate workspace and zero all fields */
> | cstate = (CopyFromStateData *) palloc0(sizeof(CopyFromStateData));
>
>
> > + Assert(!cstate->escontext->details_wanted);
>
> I'm not sure this is necessary, considering we're going to add other
> options like 'table' and 'log', which need details_wanted soon.
>
>
> --
> Regards,
make save_error_to option cannot be used with COPY TO.
add redundant test, save_error_to with COPY TO test.
Attachments:
[application/octet-stream] v2-0001-minor-refactor.no-cfbot (3.4K, ../../CACJufxEjYyhz3qSx0win6U4ZvJB7R5eWRSBau8oAyq3xsCknSA@mail.gmail.com/2-v2-0001-minor-refactor.no-cfbot)
download
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-14 01:30 ` Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-14 01:30 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: torikoshia <[email protected]>; [email protected]; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi!
I think this is a demanding and long-waited feature. The thread is
pretty long, but mostly it was disputes about how to save the errors.
The present patch includes basic infrastructure and ability to ignore
errors, thus it's pretty simple.
On Sat, Jan 13, 2024 at 4:20 PM jian he <[email protected]> wrote:
> On Fri, Jan 12, 2024 at 10:59 AM torikoshia <[email protected]> wrote:
> >
> >
> > Thanks for reviewing!
> >
> > Updated the patch merging your suggestions except below points:
> >
> > > + cstate->num_errors = 0;
> >
> > Since cstate is already initialized in below lines, this may be
> > redundant.
> >
> > | /* Allocate workspace and zero all fields */
> > | cstate = (CopyFromStateData *) palloc0(sizeof(CopyFromStateData));
> >
> >
> > > + Assert(!cstate->escontext->details_wanted);
> >
> > I'm not sure this is necessary, considering we're going to add other
> > options like 'table' and 'log', which need details_wanted soon.
> >
> >
> > --
> > Regards,
>
> make save_error_to option cannot be used with COPY TO.
> add redundant test, save_error_to with COPY TO test.
I've incorporated these changes. Also, I've changed
CopyFormatOptions.save_error_to to enum and made some edits in
comments and the commit message. I'm going to push this if there are
no objections.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Add-new-COPY-option-SAVE_ERROR_TO-v3.patch (17.7K, ../../CAPpHfdta7UJWkKVYnNwdKiYGfi4EJtXApDb7WdreLvc4jdfPgQ@mail.gmail.com/2-0001-Add-new-COPY-option-SAVE_ERROR_TO-v3.patch)
download | inline diff:
From c6033d4330e86bd33f90d36eb75b2c0427a54d82 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 14 Jan 2024 02:09:32 +0200
Subject: [PATCH] Add new COPY option SAVE_ERROR_TO
Currently, when source data contains unexpected data regarding data type or
range, the entire COPY fails. However, in some cases, such data can be ignored
and just copying normal data is preferable.
This commit adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.
Currently, SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He
Discussion: https://postgr.es/m/87k31ftoe0.fsf_-_%40commandprompt.com
Reviewed-by: Pavel Stehule, Andres Freund, Tom Lane, Daniel Gustafsson,
Reviewed-by: Alena Rybakina, Andy Fan, Andrei Lepikhov, Masahiko Sawada
Reviewed-by: Vignesh C
---
doc/src/sgml/ref/copy.sgml | 21 ++++++++++-
src/backend/commands/copy.c | 25 +++++++++++++
src/backend/commands/copyfrom.c | 46 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 17 ++++++---
src/bin/psql/tab-complete.c | 7 +++-
src/include/commands/copy.h | 11 ++++++
src/include/commands/copyfrom_internal.h | 5 +++
src/test/regress/expected/copy2.out | 36 +++++++++++++++++++
src/test/regress/sql/copy2.sql | 29 +++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 191 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index e2ffbbdf84e..e15d5a621b8 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,23 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies to save error information to <replaceable class="parameter">
+ location</replaceable> when there is malformed data in the input.
+ If this option is specified, <command>COPY</command> skips malformed data
+ and continues copying data.
+ Currently, only the <literal>none</literal> value is supported.
+ If this option is omitted, <command>COPY</command> stops operation at the
+ first error.
+ This option is allowed only in <command>COPY FROM</command>, and only when
+ not using <literal>binary</literal> format.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +574,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d77..8fc54e028a3 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -571,6 +571,26 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ char *location = defGetString(defel);
+
+ if (opts_out->save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ errorConflictingDefElem(defel, pstate);
+ else if (strcmp(location, "none") == 0)
+ opts_out->save_error_to = COPY_SAVE_ERROR_TO_NONE;
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", location),
+ parser_errposition(pstate, defel->location)));
+
+ if (!is_from)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY SAVE_ERROR_TO cannot be used with COPY TO"),
+ parser_errposition(pstate, defel->location)));
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +618,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c7..be6a151528e 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -656,6 +657,9 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ Assert(cstate->escontext);
+
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +996,25 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->escontext->error_occurred)
+ {
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+
+ /*
+ * Just make ErrorSaveContext ready for the next NextCopyFrom.
+ * Since we don't set details_wanted and error_data is not to
+ * be filled, just resetting error_occurred is enough.
+ */
+ cstate->escontext->error_occurred = false;
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1281,6 +1304,12 @@ CopyFrom(CopyFromState cstate)
CopyMultiInsertInfoFlush(&multiInsertInfo, NULL, &processed);
}
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->num_errors > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors));
+
/* Done, clean up */
error_context_stack = errcallback.previous;
@@ -1419,6 +1448,23 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ {
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->error_occurred = false;
+
+ /*
+ * Currently we only support COPY_SAVE_ERROR_TO_NONE. We'll add other
+ * options later
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ cstate->escontext->details_wanted = false;
+ }
+ else
+ cstate->escontext = NULL;
+
/* Convert FORCE_NULL name list to per-column flags, check validity */
cstate->opts.force_null_flags = (bool *) palloc0(num_phys_attrs * sizeof(bool));
if (cstate->opts.force_null_all)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f6450..7207eb26983 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -955,11 +956,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ else if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e42..efe2b7cc101 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2fc..7d1a6286a6f 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -30,6 +30,16 @@ typedef enum CopyHeaderChoice
COPY_HEADER_MATCH,
} CopyHeaderChoice;
+/*
+ * Represents where to save input processing errors. More values to be added
+ * in the future.
+ */
+typedef enum CopySaveErrorToChoice
+{
+ COPY_SAVE_ERROR_TO_UNSPECIFIED = 0, /* immediately throw errors */
+ COPY_SAVE_ERROR_TO_NONE, /* ignore errors */
+} CopySaveErrorToChoice;
+
/*
* A struct to hold COPY options, in a parsed form. All of these are related
* to formatting, except for 'freeze', which doesn't really belong here, but
@@ -62,6 +72,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ CopySaveErrorToChoice save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a9071..cad52fcc783 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,10 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions
+ * execution */
+ uint64 num_errors; /* total number of rows which contained soft
+ * errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07c..97fea200310 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -77,11 +77,21 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
ERROR: conflicting or redundant options
LINE 1: COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii...
^
+COPY x from stdin (save_error_to none,save_error_to none);
+ERROR: conflicting or redundant options
+LINE 1: COPY x from stdin (save_error_to none,save_error_to none);
+ ^
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x from stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
+COPY x to stdin (save_error_to none);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (save_error_to none);
+ ^
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +104,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY save_error_to "unsupported" not recognized
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +724,26 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+WARNING: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +758,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60867..fda46f86c9e 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -66,16 +66,20 @@ COPY x from stdin (force_not_null (a), force_not_null (b));
COPY x from stdin (force_null (a), force_null (b));
COPY x from stdin (convert_selectively (a), convert_selectively (b));
COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
+COPY x from stdin (save_error_to none,save_error_to none);
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x from stdin (format BINARY, save_error_to none);
+COPY x to stdin (save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +498,29 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +535,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..29fd1cae641 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+CopySaveErrorToChoice
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-14 20:34 ` Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-01-14 20:34 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: jian he <[email protected]>; torikoshia <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Sun, Jan 14, 2024 at 10:30 AM Alexander Korotkov
<[email protected]> wrote:
>
> Hi!
>
> I think this is a demanding and long-waited feature. The thread is
> pretty long, but mostly it was disputes about how to save the errors.
> The present patch includes basic infrastructure and ability to ignore
> errors, thus it's pretty simple.
>
> On Sat, Jan 13, 2024 at 4:20 PM jian he <[email protected]> wrote:
> > On Fri, Jan 12, 2024 at 10:59 AM torikoshia <[email protected]> wrote:
> > >
> > >
> > > Thanks for reviewing!
> > >
> > > Updated the patch merging your suggestions except below points:
> > >
> > > > + cstate->num_errors = 0;
> > >
> > > Since cstate is already initialized in below lines, this may be
> > > redundant.
> > >
> > > | /* Allocate workspace and zero all fields */
> > > | cstate = (CopyFromStateData *) palloc0(sizeof(CopyFromStateData));
> > >
> > >
> > > > + Assert(!cstate->escontext->details_wanted);
> > >
> > > I'm not sure this is necessary, considering we're going to add other
> > > options like 'table' and 'log', which need details_wanted soon.
> > >
> > >
> > > --
> > > Regards,
> >
> > make save_error_to option cannot be used with COPY TO.
> > add redundant test, save_error_to with COPY TO test.
>
> I've incorporated these changes. Also, I've changed
> CopyFormatOptions.save_error_to to enum and made some edits in
> comments and the commit message. I'm going to push this if there are
> no objections.
Thank you for updating the patch. Here are two comments:
---
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->num_errors > 0)
+ ereport(WARNING,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors));
+
/* Done, clean up */
error_context_stack = errcallback.previous;
If a malformed input is not the last data, the context message seems odd:
postgres(1:1769258)=# create table test (a int);
CREATE TABLE
postgres(1:1769258)=# copy test from stdin (save_error_to none);
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> a
>> 1
>>
2024-01-15 05:05:53.980 JST [1769258] WARNING: 1 rows were skipped
due to data type incompatibility
2024-01-15 05:05:53.980 JST [1769258] CONTEXT: COPY test, line 3: ""
COPY 1
I think it's better to report the WARNING after resetting the
error_context_stack. Or is a WARNING really appropriate here? The
v15-0001-Make-COPY-FROM-more-error-tolerant.patch[1] uses NOTICE but
the v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch[2] changes it to
WARNING without explanation.
---
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
We might want to cover the extra data cases too.
Regards,
[1] https://www.postgresql.org/message-id/CACJufxEkkqnozdnvNMGxVAA94KZaCPkYw_Cx4JKG9ueNaZma_A%40mail.gma...
[2] https://www.postgresql.org/message-id/3d0b349ddbd4ae5f605f77b491697158%40oss.nttdata.com
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-01-14 23:21 ` Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-14 23:21 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: jian he <[email protected]>; torikoshia <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Sun, Jan 14, 2024 at 10:35 PM Masahiko Sawada <[email protected]> wrote:
> Thank you for updating the patch. Here are two comments:
>
> ---
> + if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
> + cstate->num_errors > 0)
> + ereport(WARNING,
> + errmsg("%zd rows were skipped due to data type incompatibility",
> + cstate->num_errors));
> +
> /* Done, clean up */
> error_context_stack = errcallback.previous;
>
> If a malformed input is not the last data, the context message seems odd:
>
> postgres(1:1769258)=# create table test (a int);
> CREATE TABLE
> postgres(1:1769258)=# copy test from stdin (save_error_to none);
> Enter data to be copied followed by a newline.
> End with a backslash and a period on a line by itself, or an EOF signal.
> >> a
> >> 1
> >>
> 2024-01-15 05:05:53.980 JST [1769258] WARNING: 1 rows were skipped
> due to data type incompatibility
> 2024-01-15 05:05:53.980 JST [1769258] CONTEXT: COPY test, line 3: ""
> COPY 1
>
> I think it's better to report the WARNING after resetting the
> error_context_stack. Or is a WARNING really appropriate here? The
> v15-0001-Make-COPY-FROM-more-error-tolerant.patch[1] uses NOTICE but
> the v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch[2] changes it to
> WARNING without explanation.
Thank you for noticing this. I think NOTICE is more appropriate here.
There is nothing to "worry" about: the user asked to ignore the errors
and we did. And yes, it doesn't make sense to use the last line as
the context. Fixed.
> ---
> +-- test missing data: should fail
> +COPY check_ign_err FROM STDIN WITH (save_error_to none);
> +1 {1}
> +\.
>
> We might want to cover the extra data cases too.
Agreed, the relevant test is added.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Add-new-COPY-option-SAVE_ERROR_TO-v4.patch (18.0K, ../../CAPpHfdscUMTo8uzoJKj7bzCeSnus0528dPXn8=-nxp9YG3nNYw@mail.gmail.com/2-0001-Add-new-COPY-option-SAVE_ERROR_TO-v4.patch)
download | inline diff:
From 26ac277594a0fd6853c9b09afa10bf56e9f2818b Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 14 Jan 2024 02:09:32 +0200
Subject: [PATCH] Add new COPY option SAVE_ERROR_TO
Currently, when source data contains unexpected data regarding data type or
range, the entire COPY fails. However, in some cases, such data can be ignored
and just copying normal data is preferable.
This commit adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.
Currently, SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He
Discussion: https://postgr.es/m/87k31ftoe0.fsf_-_%40commandprompt.com
Reviewed-by: Pavel Stehule, Andres Freund, Tom Lane, Daniel Gustafsson,
Reviewed-by: Alena Rybakina, Andy Fan, Andrei Lepikhov, Masahiko Sawada
Reviewed-by: Vignesh C
---
doc/src/sgml/ref/copy.sgml | 21 ++++++++++-
src/backend/commands/copy.c | 25 +++++++++++++
src/backend/commands/copyfrom.c | 46 ++++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 17 ++++++---
src/bin/psql/tab-complete.c | 7 +++-
src/include/commands/copy.h | 11 ++++++
src/include/commands/copyfrom_internal.h | 5 +++
src/test/regress/expected/copy2.out | 40 +++++++++++++++++++++
src/test/regress/sql/copy2.sql | 34 ++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 200 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index e2ffbbdf84e..e15d5a621b8 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,23 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies to save error information to <replaceable class="parameter">
+ location</replaceable> when there is malformed data in the input.
+ If this option is specified, <command>COPY</command> skips malformed data
+ and continues copying data.
+ Currently, only the <literal>none</literal> value is supported.
+ If this option is omitted, <command>COPY</command> stops operation at the
+ first error.
+ This option is allowed only in <command>COPY FROM</command>, and only when
+ not using <literal>binary</literal> format.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +574,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d77..8fc54e028a3 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -571,6 +571,26 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ char *location = defGetString(defel);
+
+ if (opts_out->save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ errorConflictingDefElem(defel, pstate);
+ else if (strcmp(location, "none") == 0)
+ opts_out->save_error_to = COPY_SAVE_ERROR_TO_NONE;
+ else
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", location),
+ parser_errposition(pstate, defel->location)));
+
+ if (!is_from)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY SAVE_ERROR_TO cannot be used with COPY TO"),
+ parser_errposition(pstate, defel->location)));
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +618,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c7..d86c24e3140 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -656,6 +657,9 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ Assert(cstate->escontext);
+
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +996,25 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->escontext->error_occurred)
+ {
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+
+ /*
+ * Just make ErrorSaveContext ready for the next NextCopyFrom.
+ * Since we don't set details_wanted and error_data is not to
+ * be filled, just resetting error_occurred is enough.
+ */
+ cstate->escontext->error_occurred = false;
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1284,6 +1307,12 @@ CopyFrom(CopyFromState cstate)
/* Done, clean up */
error_context_stack = errcallback.previous;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->num_errors > 0)
+ ereport(NOTICE,
+ errmsg("%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors));
+
if (bistate != NULL)
FreeBulkInsertState(bistate);
@@ -1419,6 +1448,23 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED)
+ {
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->error_occurred = false;
+
+ /*
+ * Currently we only support COPY_SAVE_ERROR_TO_NONE. We'll add other
+ * options later
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ cstate->escontext->details_wanted = false;
+ }
+ else
+ cstate->escontext = NULL;
+
/* Convert FORCE_NULL name list to per-column flags, check validity */
cstate->opts.force_null_flags = (bool *) palloc0(num_phys_attrs * sizeof(bool));
if (cstate->opts.force_null_all)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f6450..7207eb26983 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -955,11 +956,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ else if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e42..efe2b7cc101 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2fc..7d1a6286a6f 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -30,6 +30,16 @@ typedef enum CopyHeaderChoice
COPY_HEADER_MATCH,
} CopyHeaderChoice;
+/*
+ * Represents where to save input processing errors. More values to be added
+ * in the future.
+ */
+typedef enum CopySaveErrorToChoice
+{
+ COPY_SAVE_ERROR_TO_UNSPECIFIED = 0, /* immediately throw errors */
+ COPY_SAVE_ERROR_TO_NONE, /* ignore errors */
+} CopySaveErrorToChoice;
+
/*
* A struct to hold COPY options, in a parsed form. All of these are related
* to formatting, except for 'freeze', which doesn't really belong here, but
@@ -62,6 +72,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ CopySaveErrorToChoice save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a9071..cad52fcc783 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,10 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions
+ * execution */
+ uint64 num_errors; /* total number of rows which contained soft
+ * errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07c..100fbf1dd1a 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -77,11 +77,21 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
ERROR: conflicting or redundant options
LINE 1: COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii...
^
+COPY x from stdin (save_error_to none,save_error_to none);
+ERROR: conflicting or redundant options
+LINE 1: COPY x from stdin (save_error_to none,save_error_to none);
+ ^
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x from stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
+COPY x to stdin (save_error_to none);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (save_error_to none);
+ ^
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +104,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY save_error_to "unsupported" not recognized
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +724,30 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+NOTICE: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: extra data after last expected column
+CONTEXT: COPY check_ign_err, line 1: "1 {1} 3 abc"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +762,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60867..f3c24647d4a 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -66,16 +66,20 @@ COPY x from stdin (force_not_null (a), force_not_null (b));
COPY x from stdin (force_null (a), force_null (b));
COPY x from stdin (convert_selectively (a), convert_selectively (b));
COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
+COPY x from stdin (save_error_to none,save_error_to none);
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x from stdin (format BINARY, save_error_to none);
+COPY x to stdin (save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +498,34 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 3 abc
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +540,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..29fd1cae641 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+CopySaveErrorToChoice
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-15 06:43 ` Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-01-15 06:43 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: jian he <[email protected]>; torikoshia <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Jan 15, 2024 at 8:21 AM Alexander Korotkov <[email protected]> wrote:
>
> On Sun, Jan 14, 2024 at 10:35 PM Masahiko Sawada <[email protected]> wrote:
> > Thank you for updating the patch. Here are two comments:
> >
> > ---
> > + if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
> > + cstate->num_errors > 0)
> > + ereport(WARNING,
> > + errmsg("%zd rows were skipped due to data type incompatibility",
> > + cstate->num_errors));
> > +
> > /* Done, clean up */
> > error_context_stack = errcallback.previous;
> >
> > If a malformed input is not the last data, the context message seems odd:
> >
> > postgres(1:1769258)=# create table test (a int);
> > CREATE TABLE
> > postgres(1:1769258)=# copy test from stdin (save_error_to none);
> > Enter data to be copied followed by a newline.
> > End with a backslash and a period on a line by itself, or an EOF signal.
> > >> a
> > >> 1
> > >>
> > 2024-01-15 05:05:53.980 JST [1769258] WARNING: 1 rows were skipped
> > due to data type incompatibility
> > 2024-01-15 05:05:53.980 JST [1769258] CONTEXT: COPY test, line 3: ""
> > COPY 1
> >
> > I think it's better to report the WARNING after resetting the
> > error_context_stack. Or is a WARNING really appropriate here? The
> > v15-0001-Make-COPY-FROM-more-error-tolerant.patch[1] uses NOTICE but
> > the v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch[2] changes it to
> > WARNING without explanation.
>
> Thank you for noticing this. I think NOTICE is more appropriate here.
> There is nothing to "worry" about: the user asked to ignore the errors
> and we did. And yes, it doesn't make sense to use the last line as
> the context. Fixed.
>
> > ---
> > +-- test missing data: should fail
> > +COPY check_ign_err FROM STDIN WITH (save_error_to none);
> > +1 {1}
> > +\.
> >
> > We might want to cover the extra data cases too.
>
> Agreed, the relevant test is added.
Thank you for updating the patch. I have one minor point:
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
+ cstate->num_errors > 0)
+ ereport(NOTICE,
+ errmsg("%zd rows were skipped due to
data type incompatibility",
+ cstate->num_errors));
+
We can use errmsg_plural() instead.
I have a question about the option values; do you think we need to
have another value of SAVE_ERROR_TO option to explicitly specify the
current default behavior, i.e. not accept any error? With the v4
patch, the user needs to omit SAVE_ERROR_TO option to accept errors
during COPY FROM. If we change the default behavior in the future,
many users will be affected and probably end up changing their
applications to keep the current default behavior.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-01-15 15:17 ` Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-15 15:17 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: jian he <[email protected]>; torikoshia <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Jan 15, 2024 at 8:44 AM Masahiko Sawada <[email protected]> wrote:
>
> On Mon, Jan 15, 2024 at 8:21 AM Alexander Korotkov <[email protected]> wrote:
> >
> > On Sun, Jan 14, 2024 at 10:35 PM Masahiko Sawada <[email protected]> wrote:
> > > Thank you for updating the patch. Here are two comments:
> > >
> > > ---
> > > + if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
> > > + cstate->num_errors > 0)
> > > + ereport(WARNING,
> > > + errmsg("%zd rows were skipped due to data type incompatibility",
> > > + cstate->num_errors));
> > > +
> > > /* Done, clean up */
> > > error_context_stack = errcallback.previous;
> > >
> > > If a malformed input is not the last data, the context message seems odd:
> > >
> > > postgres(1:1769258)=# create table test (a int);
> > > CREATE TABLE
> > > postgres(1:1769258)=# copy test from stdin (save_error_to none);
> > > Enter data to be copied followed by a newline.
> > > End with a backslash and a period on a line by itself, or an EOF signal.
> > > >> a
> > > >> 1
> > > >>
> > > 2024-01-15 05:05:53.980 JST [1769258] WARNING: 1 rows were skipped
> > > due to data type incompatibility
> > > 2024-01-15 05:05:53.980 JST [1769258] CONTEXT: COPY test, line 3: ""
> > > COPY 1
> > >
> > > I think it's better to report the WARNING after resetting the
> > > error_context_stack. Or is a WARNING really appropriate here? The
> > > v15-0001-Make-COPY-FROM-more-error-tolerant.patch[1] uses NOTICE but
> > > the v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch[2] changes it to
> > > WARNING without explanation.
> >
> > Thank you for noticing this. I think NOTICE is more appropriate here.
> > There is nothing to "worry" about: the user asked to ignore the errors
> > and we did. And yes, it doesn't make sense to use the last line as
> > the context. Fixed.
> >
> > > ---
> > > +-- test missing data: should fail
> > > +COPY check_ign_err FROM STDIN WITH (save_error_to none);
> > > +1 {1}
> > > +\.
> > >
> > > We might want to cover the extra data cases too.
> >
> > Agreed, the relevant test is added.
>
> Thank you for updating the patch. I have one minor point:
>
> + if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
> + cstate->num_errors > 0)
> + ereport(NOTICE,
> + errmsg("%zd rows were skipped due to
> data type incompatibility",
> + cstate->num_errors));
> +
>
> We can use errmsg_plural() instead.
Makes sense. Fixed.
> I have a question about the option values; do you think we need to
> have another value of SAVE_ERROR_TO option to explicitly specify the
> current default behavior, i.e. not accept any error? With the v4
> patch, the user needs to omit SAVE_ERROR_TO option to accept errors
> during COPY FROM. If we change the default behavior in the future,
> many users will be affected and probably end up changing their
> applications to keep the current default behavior.
Valid point. I've implemented the handling of CopySaveErrorToChoice
in a similar way to CopyHeaderChoice.
Please, check the revised patch attached.
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Add-new-COPY-option-SAVE_ERROR_TO-v5.patch (19.4K, ../../CAPpHfds2MUFKBBqzeM8aX6jpYCCgb8uiXe-5x0iQVTC=Wc8r7Q@mail.gmail.com/2-0001-Add-new-COPY-option-SAVE_ERROR_TO-v5.patch)
download | inline diff:
From 0e01ab7b1a59ca0a54ce03c482890216f43793d1 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 14 Jan 2024 02:09:32 +0200
Subject: [PATCH] Add new COPY option SAVE_ERROR_TO
Currently, when source data contains unexpected data regarding data type or
range, the entire COPY fails. However, in some cases, such data can be ignored
and just copying normal data is preferable.
This commit adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.
Currently, SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He
Discussion: https://postgr.es/m/87k31ftoe0.fsf_-_%40commandprompt.com
Reviewed-by: Pavel Stehule, Andres Freund, Tom Lane, Daniel Gustafsson,
Reviewed-by: Alena Rybakina, Andy Fan, Andrei Lepikhov, Masahiko Sawada
Reviewed-by: Vignesh C
---
doc/src/sgml/ref/copy.sgml | 23 ++++++++++-
src/backend/commands/copy.c | 49 ++++++++++++++++++++++++
src/backend/commands/copyfrom.c | 48 +++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 17 +++++---
src/bin/psql/tab-complete.c | 7 +++-
src/include/commands/copy.h | 11 ++++++
src/include/commands/copyfrom_internal.h | 5 +++
src/test/regress/expected/copy2.out | 43 +++++++++++++++++++++
src/test/regress/sql/copy2.sql | 42 ++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 239 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index e2ffbbdf84e..85881ca0ad6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,25 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies to save error information to <replaceable class="parameter">
+ location</replaceable> when there is malformed data in the input.
+ Currently, only <literal>error</literal> (default) and <literal>none</literal>
+ values are supported.
+ If the <literal>error</literal> value is specified,
+ <command>COPY</command> stops operation at the first error.
+ If the <literal>none</literal> value is specified,
+ <command>COPY</command> skips malformed data and continues copying data.
+ The option is allowed only in <command>COPY FROM</command>.
+ The <literal>none</literal> value is allowed only when
+ not using <literal>binary</literal> format.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +576,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d77..38c00379629 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -394,6 +394,42 @@ defGetCopyHeaderChoice(DefElem *def, bool is_from)
return COPY_HEADER_FALSE; /* keep compiler quiet */
}
+/*
+ * Extract a defGetCopySaveErrorToChoice value from a DefElem.
+ */
+static CopySaveErrorToChoice
+defGetCopySaveErrorToChoice(DefElem *def, ParseState *pstate, bool is_from)
+{
+ char *sval;
+
+ if (!is_from)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY SAVE_ERROR_TO cannot be used with COPY TO"),
+ parser_errposition(pstate, def->location)));
+
+ /*
+ * If no parameter value given, assume the default value.
+ */
+ if (def->arg == NULL)
+ return COPY_SAVE_ERROR_TO_ERROR;
+
+ /*
+ * Allow "error", or "none" values.
+ */
+ sval = defGetString(def);
+ if (pg_strcasecmp(sval, "error") == 0)
+ return COPY_SAVE_ERROR_TO_ERROR;
+ if (pg_strcasecmp(sval, "none") == 0)
+ return COPY_SAVE_ERROR_TO_NONE;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", sval),
+ parser_errposition(pstate, def->location)));
+ return COPY_SAVE_ERROR_TO_ERROR; /* keep compiler quiet */
+}
+
/*
* Process the statement option list for COPY.
*
@@ -419,6 +455,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_to_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -571,6 +608,13 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ if (save_error_to_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_to_specified = true;
+ opts_out->save_error_to = defGetCopySaveErrorToChoice(defel, pstate, is_from);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +642,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c7..46b23e345b8 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -656,6 +657,9 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ Assert(cstate->escontext);
+
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +996,25 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ cstate->escontext->error_occurred)
+ {
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+
+ /*
+ * Just make ErrorSaveContext ready for the next NextCopyFrom.
+ * Since we don't set details_wanted and error_data is not to
+ * be filled, just resetting error_occurred is enough.
+ */
+ cstate->escontext->error_occurred = false;
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1284,6 +1307,14 @@ CopyFrom(CopyFromState cstate)
/* Done, clean up */
error_context_stack = errcallback.previous;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ cstate->num_errors > 0)
+ ereport(NOTICE,
+ errmsg_plural("%zd row were skipped due to data type incompatibility",
+ "%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors,
+ cstate->num_errors));
+
if (bistate != NULL)
FreeBulkInsertState(bistate);
@@ -1419,6 +1450,23 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ {
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->error_occurred = false;
+
+ /*
+ * Currently we only support COPY_SAVE_ERROR_TO_NONE. We'll add other
+ * options later
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ cstate->escontext->details_wanted = false;
+ }
+ else
+ cstate->escontext = NULL;
+
/* Convert FORCE_NULL name list to per-column flags, check validity */
cstate->opts.force_null_flags = (bool *) palloc0(num_phys_attrs * sizeof(bool));
if (cstate->opts.force_null_all)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f6450..7207eb26983 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -955,11 +956,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ else if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e42..6bfdb5f0082 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("error", "none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2fc..8972c6180d7 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -30,6 +30,16 @@ typedef enum CopyHeaderChoice
COPY_HEADER_MATCH,
} CopyHeaderChoice;
+/*
+ * Represents where to save input processing errors. More values to be added
+ * in the future.
+ */
+typedef enum CopySaveErrorToChoice
+{
+ COPY_SAVE_ERROR_TO_ERROR = 0, /* immediately throw errors */
+ COPY_SAVE_ERROR_TO_NONE, /* ignore errors */
+} CopySaveErrorToChoice;
+
/*
* A struct to hold COPY options, in a parsed form. All of these are related
* to formatting, except for 'freeze', which doesn't really belong here, but
@@ -62,6 +72,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ CopySaveErrorToChoice save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a9071..cad52fcc783 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,10 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions
+ * execution */
+ uint64 num_errors; /* total number of rows which contained soft
+ * errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07c..42cbcb2e92f 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -77,11 +77,21 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
ERROR: conflicting or redundant options
LINE 1: COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii...
^
+COPY x from stdin (save_error_to none,save_error_to none);
+ERROR: conflicting or redundant options
+LINE 1: COPY x from stdin (save_error_to none,save_error_to none);
+ ^
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x from stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
+COPY x to stdin (save_error_to none);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (save_error_to none);
+ ^
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +104,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +724,33 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to error);
+ERROR: invalid input syntax for type integer: "a"
+CONTEXT: COPY check_ign_err, line 2, column n: "a"
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+NOTICE: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: extra data after last expected column
+CONTEXT: COPY check_ign_err, line 1: "1 {1} 3 abc"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +765,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60867..c48d556350d 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -66,16 +66,20 @@ COPY x from stdin (force_not_null (a), force_not_null (b));
COPY x from stdin (force_null (a), force_null (b));
COPY x from stdin (convert_selectively (a), convert_selectively (b));
COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
+COPY x from stdin (save_error_to none,save_error_to none);
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x from stdin (format BINARY, save_error_to none);
+COPY x to stdin (save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +498,42 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to error);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 3 abc
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +548,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..29fd1cae641 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+CopySaveErrorToChoice
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-16 00:27 ` torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-01-16 00:27 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; jian he <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On 2024-01-16 00:17, Alexander Korotkov wrote:
> On Mon, Jan 15, 2024 at 8:44 AM Masahiko Sawada <[email protected]>
> wrote:
>>
>> On Mon, Jan 15, 2024 at 8:21 AM Alexander Korotkov
>> <[email protected]> wrote:
>> >
>> > On Sun, Jan 14, 2024 at 10:35 PM Masahiko Sawada <[email protected]> wrote:
>> > > Thank you for updating the patch. Here are two comments:
>> > >
>> > > ---
>> > > + if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_UNSPECIFIED &&
>> > > + cstate->num_errors > 0)
>> > > + ereport(WARNING,
>> > > + errmsg("%zd rows were skipped due to data type incompatibility",
>> > > + cstate->num_errors));
>> > > +
>> > > /* Done, clean up */
>> > > error_context_stack = errcallback.previous;
>> > >
>> > > If a malformed input is not the last data, the context message seems odd:
>> > >
>> > > postgres(1:1769258)=# create table test (a int);
>> > > CREATE TABLE
>> > > postgres(1:1769258)=# copy test from stdin (save_error_to none);
>> > > Enter data to be copied followed by a newline.
>> > > End with a backslash and a period on a line by itself, or an EOF signal.
>> > > >> a
>> > > >> 1
>> > > >>
>> > > 2024-01-15 05:05:53.980 JST [1769258] WARNING: 1 rows were skipped
>> > > due to data type incompatibility
>> > > 2024-01-15 05:05:53.980 JST [1769258] CONTEXT: COPY test, line 3: ""
>> > > COPY 1
>> > >
>> > > I think it's better to report the WARNING after resetting the
>> > > error_context_stack. Or is a WARNING really appropriate here? The
>> > > v15-0001-Make-COPY-FROM-more-error-tolerant.patch[1] uses NOTICE but
>> > > the v1-0001-Add-new-COPY-option-SAVE_ERROR_TO.patch[2] changes it to
>> > > WARNING without explanation.
>> >
>> > Thank you for noticing this. I think NOTICE is more appropriate here.
>> > There is nothing to "worry" about: the user asked to ignore the errors
>> > and we did. And yes, it doesn't make sense to use the last line as
>> > the context. Fixed.
>> >
>> > > ---
>> > > +-- test missing data: should fail
>> > > +COPY check_ign_err FROM STDIN WITH (save_error_to none);
>> > > +1 {1}
>> > > +\.
>> > >
>> > > We might want to cover the extra data cases too.
>> >
>> > Agreed, the relevant test is added.
>>
>> Thank you for updating the patch. I have one minor point:
>>
>> + if (cstate->opts.save_error_to !=
>> COPY_SAVE_ERROR_TO_UNSPECIFIED &&
>> + cstate->num_errors > 0)
>> + ereport(NOTICE,
>> + errmsg("%zd rows were skipped due to
>> data type incompatibility",
>> + cstate->num_errors));
>> +
>>
>> We can use errmsg_plural() instead.
>
> Makes sense. Fixed.
>
>> I have a question about the option values; do you think we need to
>> have another value of SAVE_ERROR_TO option to explicitly specify the
>> current default behavior, i.e. not accept any error? With the v4
>> patch, the user needs to omit SAVE_ERROR_TO option to accept errors
>> during COPY FROM. If we change the default behavior in the future,
>> many users will be affected and probably end up changing their
>> applications to keep the current default behavior.
>
> Valid point. I've implemented the handling of CopySaveErrorToChoice
> in a similar way to CopyHeaderChoice.
>
> Please, check the revised patch attached.
Thanks for updating the patch!
Here is a minor comment:
> +/*
> + * Extract a defGetCopySaveErrorToChoice value from a DefElem.
> + */
Should be Extract a "CopySaveErrorToChoice"?
BTW I'm thinking we should add a column to pg_stat_progress_copy that
counts soft errors. I'll suggest this in another thread.
> ------
> Regards,
> Alexander Korotkov
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-16 15:08 ` Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-16 15:08 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; jian he <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi,
> Thanks for updating the patch!
You're welcome!
> Here is a minor comment:
>
> > +/*
> > + * Extract a defGetCopySaveErrorToChoice value from a DefElem.
> > + */
>
> Should be Extract a "CopySaveErrorToChoice"?
Fixed.
> BTW I'm thinking we should add a column to pg_stat_progress_copy that
> counts soft errors. I'll suggest this in another thread.
Please do!
------
Regards,
Alexander Korotkov
Attachments:
[application/octet-stream] 0001-Add-new-COPY-option-SAVE_ERROR_TO-v6.patch (19.4K, ../../CAPpHfdvfpON-osWgDg-gsJGi9MnC4XdAWCdcH1vJcq9LOa97cw@mail.gmail.com/2-0001-Add-new-COPY-option-SAVE_ERROR_TO-v6.patch)
download | inline diff:
From d7d31a4ac6b5e4002995a9e2445bd425aa9e0fbd Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Sun, 14 Jan 2024 02:09:32 +0200
Subject: [PATCH] Add new COPY option SAVE_ERROR_TO
Currently, when source data contains unexpected data regarding data type or
range, the entire COPY fails. However, in some cases, such data can be ignored
and just copying normal data is preferable.
This commit adds a new option SAVE_ERROR_TO, which specifies where to save the
error information. When this option is specified, COPY skips soft errors and
continues copying.
Currently, SAVE_ERROR_TO only supports "none". This indicates error information
is not saved and COPY just skips the unexpected data and continues running.
Later works are expected to add more choices, such as 'log' and 'table'.
Author: Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He
Discussion: https://postgr.es/m/87k31ftoe0.fsf_-_%40commandprompt.com
Reviewed-by: Pavel Stehule, Andres Freund, Tom Lane, Daniel Gustafsson,
Reviewed-by: Alena Rybakina, Andy Fan, Andrei Lepikhov, Masahiko Sawada
Reviewed-by: Vignesh C
---
doc/src/sgml/ref/copy.sgml | 23 ++++++++++-
src/backend/commands/copy.c | 49 ++++++++++++++++++++++++
src/backend/commands/copyfrom.c | 48 +++++++++++++++++++++++
src/backend/commands/copyfromparse.c | 17 +++++---
src/bin/psql/tab-complete.c | 7 +++-
src/include/commands/copy.h | 11 ++++++
src/include/commands/copyfrom_internal.h | 5 +++
src/test/regress/expected/copy2.out | 43 +++++++++++++++++++++
src/test/regress/sql/copy2.sql | 42 ++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 239 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index e2ffbbdf84e..85881ca0ad6 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,6 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
+ SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -373,6 +374,25 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>SAVE_ERROR_TO</literal></term>
+ <listitem>
+ <para>
+ Specifies to save error information to <replaceable class="parameter">
+ location</replaceable> when there is malformed data in the input.
+ Currently, only <literal>error</literal> (default) and <literal>none</literal>
+ values are supported.
+ If the <literal>error</literal> value is specified,
+ <command>COPY</command> stops operation at the first error.
+ If the <literal>none</literal> value is specified,
+ <command>COPY</command> skips malformed data and continues copying data.
+ The option is allowed only in <command>COPY FROM</command>.
+ The <literal>none</literal> value is allowed only when
+ not using <literal>binary</literal> format.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>ENCODING</literal></term>
<listitem>
@@ -556,7 +576,8 @@ COPY <replaceable class="parameter">count</replaceable>
</para>
<para>
- <command>COPY</command> stops operation at the first error. This
+ <command>COPY</command> stops operation at the first error when
+ <literal>SAVE_ERROR_TO</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index fe4cf957d77..c36d7f1daaf 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -394,6 +394,42 @@ defGetCopyHeaderChoice(DefElem *def, bool is_from)
return COPY_HEADER_FALSE; /* keep compiler quiet */
}
+/*
+ * Extract a CopySaveErrorToChoice value from a DefElem.
+ */
+static CopySaveErrorToChoice
+defGetCopySaveErrorToChoice(DefElem *def, ParseState *pstate, bool is_from)
+{
+ char *sval;
+
+ if (!is_from)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY SAVE_ERROR_TO cannot be used with COPY TO"),
+ parser_errposition(pstate, def->location)));
+
+ /*
+ * If no parameter value given, assume the default value.
+ */
+ if (def->arg == NULL)
+ return COPY_SAVE_ERROR_TO_ERROR;
+
+ /*
+ * Allow "error", or "none" values.
+ */
+ sval = defGetString(def);
+ if (pg_strcasecmp(sval, "error") == 0)
+ return COPY_SAVE_ERROR_TO_ERROR;
+ if (pg_strcasecmp(sval, "none") == 0)
+ return COPY_SAVE_ERROR_TO_NONE;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("COPY save_error_to \"%s\" not recognized", sval),
+ parser_errposition(pstate, def->location)));
+ return COPY_SAVE_ERROR_TO_ERROR; /* keep compiler quiet */
+}
+
/*
* Process the statement option list for COPY.
*
@@ -419,6 +455,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
+ bool save_error_to_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -571,6 +608,13 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
+ else if (strcmp(defel->defname, "save_error_to") == 0)
+ {
+ if (save_error_to_specified)
+ errorConflictingDefElem(defel, pstate);
+ save_error_to_specified = true;
+ opts_out->save_error_to = defGetCopySaveErrorToChoice(defel, pstate, is_from);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -598,6 +642,11 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
+ if (opts_out->binary && opts_out->save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+
/* Set defaults for omitted options */
if (!opts_out->delim)
opts_out->delim = opts_out->csv_mode ? "," : "\t";
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 37836a769c7..46b23e345b8 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -42,6 +42,7 @@
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "rewrite/rewriteHandler.h"
@@ -656,6 +657,9 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ Assert(cstate->escontext);
+
/*
* The target must be a plain, foreign, or partitioned relation, or have
* an INSTEAD OF INSERT row trigger. (Currently, such triggers are only
@@ -992,6 +996,25 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ cstate->escontext->error_occurred)
+ {
+ /*
+ * Soft error occured, skip this tuple and save error information
+ * according to SAVE_ERROR_TO.
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+
+ /*
+ * Just make ErrorSaveContext ready for the next NextCopyFrom.
+ * Since we don't set details_wanted and error_data is not to
+ * be filled, just resetting error_occurred is enough.
+ */
+ cstate->escontext->error_occurred = false;
+
+ continue;
+ }
+
ExecStoreVirtualTuple(myslot);
/*
@@ -1284,6 +1307,14 @@ CopyFrom(CopyFromState cstate)
/* Done, clean up */
error_context_stack = errcallback.previous;
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ cstate->num_errors > 0)
+ ereport(NOTICE,
+ errmsg_plural("%zd row were skipped due to data type incompatibility",
+ "%zd rows were skipped due to data type incompatibility",
+ cstate->num_errors,
+ cstate->num_errors));
+
if (bistate != NULL)
FreeBulkInsertState(bistate);
@@ -1419,6 +1450,23 @@ BeginCopyFrom(ParseState *pstate,
}
}
+ /* Set up soft error handler for SAVE_ERROR_TO */
+ if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ {
+ cstate->escontext = makeNode(ErrorSaveContext);
+ cstate->escontext->type = T_ErrorSaveContext;
+ cstate->escontext->error_occurred = false;
+
+ /*
+ * Currently we only support COPY_SAVE_ERROR_TO_NONE. We'll add other
+ * options later
+ */
+ if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ cstate->escontext->details_wanted = false;
+ }
+ else
+ cstate->escontext = NULL;
+
/* Convert FORCE_NULL name list to per-column flags, check validity */
cstate->opts.force_null_flags = (bool *) palloc0(num_phys_attrs * sizeof(bool));
if (cstate->opts.force_null_all)
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index af4c36f6450..7207eb26983 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -70,6 +70,7 @@
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "nodes/miscnodes.h"
#include "pgstat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
@@ -955,11 +956,17 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- else
- values[m] = InputFunctionCall(&in_functions[m],
- string,
- typioparams[m],
- att->atttypmod);
+ /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ else if (!InputFunctionCallSafe(&in_functions[m],
+ string,
+ typioparams[m],
+ att->atttypmod,
+ (Node *) cstate->escontext,
+ &values[m]))
+ {
+ cstate->num_errors++;
+ return true;
+ }
cstate->cur_attname = NULL;
cstate->cur_attval = NULL;
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 09914165e42..6bfdb5f0082 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2898,12 +2898,17 @@ psql_completion(const char *text, int start, int end)
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "("))
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
- "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT");
+ "FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
+ "SAVE_ERROR_TO");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
+ /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
+ COMPLETE_WITH("error", "none");
+
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
COMPLETE_WITH("WHERE");
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index e6c1867a2fc..8972c6180d7 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -30,6 +30,16 @@ typedef enum CopyHeaderChoice
COPY_HEADER_MATCH,
} CopyHeaderChoice;
+/*
+ * Represents where to save input processing errors. More values to be added
+ * in the future.
+ */
+typedef enum CopySaveErrorToChoice
+{
+ COPY_SAVE_ERROR_TO_ERROR = 0, /* immediately throw errors */
+ COPY_SAVE_ERROR_TO_NONE, /* ignore errors */
+} CopySaveErrorToChoice;
+
/*
* A struct to hold COPY options, in a parsed form. All of these are related
* to formatting, except for 'freeze', which doesn't really belong here, but
@@ -62,6 +72,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
+ CopySaveErrorToChoice save_error_to; /* where to save error information */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h
index 715939a9071..cad52fcc783 100644
--- a/src/include/commands/copyfrom_internal.h
+++ b/src/include/commands/copyfrom_internal.h
@@ -16,6 +16,7 @@
#include "commands/copy.h"
#include "commands/trigger.h"
+#include "nodes/miscnodes.h"
/*
* Represents the different source cases we need to worry about at
@@ -94,6 +95,10 @@ typedef struct CopyFromStateData
* default value */
FmgrInfo *in_functions; /* array of input functions for each attrs */
Oid *typioparams; /* array of element types for in_functions */
+ ErrorSaveContext *escontext; /* soft error trapper during in_functions
+ * execution */
+ uint64 num_errors; /* total number of rows which contained soft
+ * errors */
int *defmap; /* array of default att numbers related to
* missing att */
ExprState **defexprs; /* array of default att expressions for all
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index c4178b9c07c..42cbcb2e92f 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -77,11 +77,21 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
ERROR: conflicting or redundant options
LINE 1: COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii...
^
+COPY x from stdin (save_error_to none,save_error_to none);
+ERROR: conflicting or redundant options
+LINE 1: COPY x from stdin (save_error_to none,save_error_to none);
+ ^
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
+COPY x from stdin (format BINARY, save_error_to none);
+ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
+COPY x to stdin (save_error_to none);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (save_error_to none);
+ ^
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -94,6 +104,10 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
+COPY x to stdin (format BINARY, save_error_to unsupported);
+ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
+LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+ ^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
ERROR: column "d" specified more than once
@@ -710,6 +724,33 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to error);
+ERROR: invalid input syntax for type integer: "a"
+CONTEXT: COPY check_ign_err, line 2, column n: "a"
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+NOTICE: 4 rows were skipped due to data type incompatibility
+SELECT * FROM check_ign_err;
+ n | m | k
+---+-----+---
+ 1 | {1} | 1
+ 5 | {5} | 5
+(2 rows)
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+ERROR: invalid input syntax for type widget: "1"
+CONTEXT: COPY hard_err, line 1, column foo: "1"
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: missing data for column "k"
+CONTEXT: COPY check_ign_err, line 1: "1 {1}"
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+ERROR: extra data after last expected column
+CONTEXT: COPY check_ign_err, line 1: "1 {1} 3 abc"
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -724,6 +765,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
--
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index a5486f60867..c48d556350d 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -66,16 +66,20 @@ COPY x from stdin (force_not_null (a), force_not_null (b));
COPY x from stdin (force_null (a), force_null (b));
COPY x from stdin (convert_selectively (a), convert_selectively (b));
COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
+COPY x from stdin (save_error_to none,save_error_to none);
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
+COPY x from stdin (format BINARY, save_error_to none);
+COPY x to stdin (save_error_to none);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
+COPY x to stdin (format BINARY, save_error_to unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -494,6 +498,42 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
+-- tests for SAVE_ERROR_TO option
+CREATE TABLE check_ign_err (n int, m int[], k int);
+COPY check_ign_err FROM STDIN WITH (save_error_to error);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 1
+a {2} 2
+3 {3} 3333333333
+4 {a, 4} 4
+
+5 {5} 5
+\.
+SELECT * FROM check_ign_err;
+
+-- test datatype error that can't be handled as soft: should fail
+CREATE TABLE hard_err(foo widget);
+COPY hard_err FROM STDIN WITH (save_error_to none);
+1
+\.
+
+-- test missing data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1}
+\.
+
+-- test extra data: should fail
+COPY check_ign_err FROM STDIN WITH (save_error_to none);
+1 {1} 3 abc
+\.
+
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
@@ -508,6 +548,8 @@ DROP TABLE instead_of_insert_tbl;
DROP VIEW instead_of_insert_tbl_view;
DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
+DROP TABLE check_ign_err;
+DROP TABLE hard_err;
--
-- COPY FROM ... DEFAULT
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f582eb59e7d..29fd1cae641 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -4041,3 +4041,4 @@ manifest_writer
rfile
ws_options
ws_file_info
+CopySaveErrorToChoice
--
2.39.3 (Apple Git-145)
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-17 05:38 ` torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:01 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 2 replies; 117+ messages in thread
From: torikoshia @ 2024-01-17 05:38 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; jian he <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
Hi,
Thanks for applying!
> + errmsg_plural("%zd row were skipped due
> to data type incompatibility",
Sorry, I just noticed it, but 'were' should be 'was' here?
>> BTW I'm thinking we should add a column to pg_stat_progress_copy that
>> counts soft errors. I'll suggest this in another thread.
> Please do!
I've started it here:
https://www.postgresql.org/message-id/[email protected]
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-17 07:48 ` Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: Kyotaro Horiguchi @ 2024-01-17 07:48 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
At Wed, 17 Jan 2024 14:38:54 +0900, torikoshia <[email protected]> wrote in
> Hi,
>
> Thanks for applying!
>
> > + errmsg_plural("%zd row were skipped due to data type
> > incompatibility",
>
> Sorry, I just noticed it, but 'were' should be 'was' here?
>
> >> BTW I'm thinking we should add a column to pg_stat_progress_copy that
> >> counts soft errors. I'll suggest this in another thread.
> > Please do!
>
> I've started it here:
>
> https://www.postgresql.org/message-id/[email protected]
Switching topics, this commit (9e2d870119) adds the following help message:
> "COPY { %s [ ( %s [, ...] ) ] | ( %s ) }\n"
> " TO { '%s' | PROGRAM '%s' | STDOUT }\n"
> ...
> " SAVE_ERROR_TO '%s'\n"
> ...
> _("location"),
On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
indicate "immediately error out" and 'just ignore the failure'
respectively, but these options hardly seem to denote a 'location',
and appear more like an 'action'. I somewhat suspect that this
parameter name intially conceived with the assupmtion that it would
take file names or similar parameters. I'm not sure if others will
agree, but I think the parameter name might not be the best
choice. For instance, considering the addition of the third value
'log', something like on_error_action (error, ignore, log) would be
more intuitively understandable. What do you think?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
@ 2024-01-17 21:06 ` Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-17 21:06 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Wed, Jan 17, 2024 at 9:49 AM Kyotaro Horiguchi
<[email protected]> wrote:
> At Wed, 17 Jan 2024 14:38:54 +0900, torikoshia <[email protected]> wrote in
> > Hi,
> >
> > Thanks for applying!
> >
> > > + errmsg_plural("%zd row were skipped due to data type
> > > incompatibility",
> >
> > Sorry, I just noticed it, but 'were' should be 'was' here?
> >
> > >> BTW I'm thinking we should add a column to pg_stat_progress_copy that
> > >> counts soft errors. I'll suggest this in another thread.
> > > Please do!
> >
> > I've started it here:
> >
> > https://www.postgresql.org/message-id/[email protected]
>
> Switching topics, this commit (9e2d870119) adds the following help message:
>
>
> > "COPY { %s [ ( %s [, ...] ) ] | ( %s ) }\n"
> > " TO { '%s' | PROGRAM '%s' | STDOUT }\n"
> > ...
> > " SAVE_ERROR_TO '%s'\n"
> > ...
> > _("location"),
>
> On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
> indicate "immediately error out" and 'just ignore the failure'
> respectively, but these options hardly seem to denote a 'location',
> and appear more like an 'action'. I somewhat suspect that this
> parameter name intially conceived with the assupmtion that it would
> take file names or similar parameters. I'm not sure if others will
> agree, but I think the parameter name might not be the best
> choice. For instance, considering the addition of the third value
> 'log', something like on_error_action (error, ignore, log) would be
> more intuitively understandable. What do you think?
Probably, but I'm not sure about that. The name SAVE_ERROR_TO assumes
the next word will be location, not action. With some stretch we can
assume 'error' to be location. I think it would be even more stretchy
to think that SAVE_ERROR_TO is followed by action. Probably, we can
replace SAVE_ERROR_TO with another name which could be naturally
followed by action, but I don't have something appropriate in mind.
However, I'm not native english speaker and certainly could miss
something.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-17 21:37 ` Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Tom Lane @ 2024-01-17 21:37 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Alexander Korotkov <[email protected]> writes:
> On Wed, Jan 17, 2024 at 9:49 AM Kyotaro Horiguchi
> <[email protected]> wrote:
>> On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
>> indicate "immediately error out" and 'just ignore the failure'
>> respectively, but these options hardly seem to denote a 'location',
>> and appear more like an 'action'. I somewhat suspect that this
>> parameter name intially conceived with the assupmtion that it would
>> take file names or similar parameters. I'm not sure if others will
>> agree, but I think the parameter name might not be the best
>> choice. For instance, considering the addition of the third value
>> 'log', something like on_error_action (error, ignore, log) would be
>> more intuitively understandable. What do you think?
> Probably, but I'm not sure about that. The name SAVE_ERROR_TO assumes
> the next word will be location, not action. With some stretch we can
> assume 'error' to be location. I think it would be even more stretchy
> to think that SAVE_ERROR_TO is followed by action.
The other problem with this terminology is that with 'none', what it
is doing is the exact opposite of "saving" the errors. I agree we
need a better name.
Kyotaro-san's suggestion isn't bad, though I might shorten it to
error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
You will need a separate parameter anyway to specify the destination
of "log", unless "none" became an illegal table name when I wasn't
looking. I don't buy that one parameter that has some special values
while other values could be names will be a good design. Moreover,
what if we want to support (say) log-to-file along with log-to-table?
Trying to distinguish a file name from a table name without any other
context seems impossible.
regards, tom lane
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2024-01-18 00:56 ` Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-01-18 00:56 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>
> Alexander Korotkov <[email protected]> writes:
> > On Wed, Jan 17, 2024 at 9:49 AM Kyotaro Horiguchi
> > <[email protected]> wrote:
> >> On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
> >> indicate "immediately error out" and 'just ignore the failure'
> >> respectively, but these options hardly seem to denote a 'location',
> >> and appear more like an 'action'. I somewhat suspect that this
> >> parameter name intially conceived with the assupmtion that it would
> >> take file names or similar parameters. I'm not sure if others will
> >> agree, but I think the parameter name might not be the best
> >> choice. For instance, considering the addition of the third value
> >> 'log', something like on_error_action (error, ignore, log) would be
> >> more intuitively understandable. What do you think?
>
> > Probably, but I'm not sure about that. The name SAVE_ERROR_TO assumes
> > the next word will be location, not action. With some stretch we can
> > assume 'error' to be location. I think it would be even more stretchy
> > to think that SAVE_ERROR_TO is followed by action.
>
> The other problem with this terminology is that with 'none', what it
> is doing is the exact opposite of "saving" the errors. I agree we
> need a better name.
Agreed.
>
> Kyotaro-san's suggestion isn't bad, though I might shorten it to
> error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> You will need a separate parameter anyway to specify the destination
> of "log", unless "none" became an illegal table name when I wasn't
> looking. I don't buy that one parameter that has some special values
> while other values could be names will be a good design. Moreover,
> what if we want to support (say) log-to-file along with log-to-table?
> Trying to distinguish a file name from a table name without any other
> context seems impossible.
I've been thinking we can add more values to this option to log errors
not only to the server logs but also to the error table (not sure
details but I imagined an error table is created for each table on
error), without an additional option for the destination name. The
values would be like error_action {error|ignore|save-logs|save-table}.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-01-18 01:10 ` jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-18 01:10 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]> wrote:
>
> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >
> > Alexander Korotkov <[email protected]> writes:
> > > On Wed, Jan 17, 2024 at 9:49 AM Kyotaro Horiguchi
> > > <[email protected]> wrote:
> > >> On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
> > >> indicate "immediately error out" and 'just ignore the failure'
> > >> respectively, but these options hardly seem to denote a 'location',
> > >> and appear more like an 'action'. I somewhat suspect that this
> > >> parameter name intially conceived with the assupmtion that it would
> > >> take file names or similar parameters. I'm not sure if others will
> > >> agree, but I think the parameter name might not be the best
> > >> choice. For instance, considering the addition of the third value
> > >> 'log', something like on_error_action (error, ignore, log) would be
> > >> more intuitively understandable. What do you think?
> >
> > > Probably, but I'm not sure about that. The name SAVE_ERROR_TO assumes
> > > the next word will be location, not action. With some stretch we can
> > > assume 'error' to be location. I think it would be even more stretchy
> > > to think that SAVE_ERROR_TO is followed by action.
> >
> > The other problem with this terminology is that with 'none', what it
> > is doing is the exact opposite of "saving" the errors. I agree we
> > need a better name.
>
> Agreed.
>
> >
> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> > You will need a separate parameter anyway to specify the destination
> > of "log", unless "none" became an illegal table name when I wasn't
> > looking. I don't buy that one parameter that has some special values
> > while other values could be names will be a good design. Moreover,
> > what if we want to support (say) log-to-file along with log-to-table?
> > Trying to distinguish a file name from a table name without any other
> > context seems impossible.
>
> I've been thinking we can add more values to this option to log errors
> not only to the server logs but also to the error table (not sure
> details but I imagined an error table is created for each table on
> error), without an additional option for the destination name. The
> values would be like error_action {error|ignore|save-logs|save-table}.
>
another idea:
on_error {error|ignore|other_future_option}
if not specified then by default ERROR.
You can also specify ERROR or IGNORE for now.
I agree, the parameter "error_action" is better than "location".
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-18 02:15 ` torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-01-18 02:15 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Alexander Korotkov <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-01-18 10:10, jian he wrote:
> On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> wrote:
>>
>> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> >
>> > Alexander Korotkov <[email protected]> writes:
>> > > On Wed, Jan 17, 2024 at 9:49 AM Kyotaro Horiguchi
>> > > <[email protected]> wrote:
>> > >> On the other hand, SAVE_ERROR_TO takes 'error' or 'none', which
>> > >> indicate "immediately error out" and 'just ignore the failure'
>> > >> respectively, but these options hardly seem to denote a 'location',
>> > >> and appear more like an 'action'. I somewhat suspect that this
>> > >> parameter name intially conceived with the assupmtion that it would
>> > >> take file names or similar parameters. I'm not sure if others will
>> > >> agree, but I think the parameter name might not be the best
>> > >> choice. For instance, considering the addition of the third value
>> > >> 'log', something like on_error_action (error, ignore, log) would be
>> > >> more intuitively understandable. What do you think?
>> >
>> > > Probably, but I'm not sure about that. The name SAVE_ERROR_TO assumes
>> > > the next word will be location, not action. With some stretch we can
>> > > assume 'error' to be location. I think it would be even more stretchy
>> > > to think that SAVE_ERROR_TO is followed by action.
>> >
>> > The other problem with this terminology is that with 'none', what it
>> > is doing is the exact opposite of "saving" the errors. I agree we
>> > need a better name.
>>
>> Agreed.
>>
>> >
>> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> > You will need a separate parameter anyway to specify the destination
>> > of "log", unless "none" became an illegal table name when I wasn't
>> > looking. I don't buy that one parameter that has some special values
>> > while other values could be names will be a good design. Moreover,
>> > what if we want to support (say) log-to-file along with log-to-table?
>> > Trying to distinguish a file name from a table name without any other
>> > context seems impossible.
>>
>> I've been thinking we can add more values to this option to log errors
>> not only to the server logs but also to the error table (not sure
>> details but I imagined an error table is created for each table on
>> error), without an additional option for the destination name. The
>> values would be like error_action {error|ignore|save-logs|save-table}.
>>
>
> another idea:
> on_error {error|ignore|other_future_option}
> if not specified then by default ERROR.
> You can also specify ERROR or IGNORE for now.
>
> I agree, the parameter "error_action" is better than "location".
I'm not sure whether error_action or on_error is better, but either way
"error_action error" and "on_error error" seems a bit odd to me.
I feel "stop" is better for both cases as Tom suggested.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-18 07:59 ` Alexander Korotkov <[email protected]>
2024-01-18 08:01 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Pavel Stehule <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 12:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 3 replies; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-18 07:59 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: jian he <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> On 2024-01-18 10:10, jian he wrote:
> > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> > wrote:
> >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> >> > You will need a separate parameter anyway to specify the destination
> >> > of "log", unless "none" became an illegal table name when I wasn't
> >> > looking. I don't buy that one parameter that has some special values
> >> > while other values could be names will be a good design. Moreover,
> >> > what if we want to support (say) log-to-file along with log-to-table?
> >> > Trying to distinguish a file name from a table name without any other
> >> > context seems impossible.
> >>
> >> I've been thinking we can add more values to this option to log errors
> >> not only to the server logs but also to the error table (not sure
> >> details but I imagined an error table is created for each table on
> >> error), without an additional option for the destination name. The
> >> values would be like error_action {error|ignore|save-logs|save-table}.
> >>
> >
> > another idea:
> > on_error {error|ignore|other_future_option}
> > if not specified then by default ERROR.
> > You can also specify ERROR or IGNORE for now.
> >
> > I agree, the parameter "error_action" is better than "location".
>
> I'm not sure whether error_action or on_error is better, but either way
> "error_action error" and "on_error error" seems a bit odd to me.
> I feel "stop" is better for both cases as Tom suggested.
OK. What about this?
on_error {stop|ignore|other_future_option}
where other_future_option might be compound like "file 'copy.log'" or
"table 'copy_log'".
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-18 08:01 ` Pavel Stehule <[email protected]>
2 siblings, 0 replies; 117+ messages in thread
From: Pavel Stehule @ 2024-01-18 08:01 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: torikoshia <[email protected]>; jian he <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
čt 18. 1. 2024 v 8:59 odesílatel Alexander Korotkov <[email protected]>
napsal:
> On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]>
> wrote:
> > On 2024-01-18 10:10, jian he wrote:
> > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]
> >
> > > wrote:
> > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> > >> > error_action {error|ignore|log} (or perhaps "stop" instead of
> "error")?
> > >> > You will need a separate parameter anyway to specify the destination
> > >> > of "log", unless "none" became an illegal table name when I wasn't
> > >> > looking. I don't buy that one parameter that has some special
> values
> > >> > while other values could be names will be a good design. Moreover,
> > >> > what if we want to support (say) log-to-file along with
> log-to-table?
> > >> > Trying to distinguish a file name from a table name without any
> other
> > >> > context seems impossible.
> > >>
> > >> I've been thinking we can add more values to this option to log errors
> > >> not only to the server logs but also to the error table (not sure
> > >> details but I imagined an error table is created for each table on
> > >> error), without an additional option for the destination name. The
> > >> values would be like error_action {error|ignore|save-logs|save-table}.
> > >>
> > >
> > > another idea:
> > > on_error {error|ignore|other_future_option}
> > > if not specified then by default ERROR.
> > > You can also specify ERROR or IGNORE for now.
> > >
> > > I agree, the parameter "error_action" is better than "location".
> >
> > I'm not sure whether error_action or on_error is better, but either way
> > "error_action error" and "on_error error" seems a bit odd to me.
> > I feel "stop" is better for both cases as Tom suggested.
>
> OK. What about this?
> on_error {stop|ignore|other_future_option}
> where other_future_option might be compound like "file 'copy.log'" or
> "table 'copy_log'".
>
+1
it is consistent with psql
Regards
Pavel
>
> ------
> Regards,
> Alexander Korotkov
>
>
>
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-18 08:33 ` Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-01-18 08:33 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: torikoshia <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov <[email protected]> wrote:
>
> On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> > On 2024-01-18 10:10, jian he wrote:
> > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> > > wrote:
> > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> > >> > You will need a separate parameter anyway to specify the destination
> > >> > of "log", unless "none" became an illegal table name when I wasn't
> > >> > looking. I don't buy that one parameter that has some special values
> > >> > while other values could be names will be a good design. Moreover,
> > >> > what if we want to support (say) log-to-file along with log-to-table?
> > >> > Trying to distinguish a file name from a table name without any other
> > >> > context seems impossible.
> > >>
> > >> I've been thinking we can add more values to this option to log errors
> > >> not only to the server logs but also to the error table (not sure
> > >> details but I imagined an error table is created for each table on
> > >> error), without an additional option for the destination name. The
> > >> values would be like error_action {error|ignore|save-logs|save-table}.
> > >>
> > >
> > > another idea:
> > > on_error {error|ignore|other_future_option}
> > > if not specified then by default ERROR.
> > > You can also specify ERROR or IGNORE for now.
> > >
> > > I agree, the parameter "error_action" is better than "location".
> >
> > I'm not sure whether error_action or on_error is better, but either way
> > "error_action error" and "on_error error" seems a bit odd to me.
> > I feel "stop" is better for both cases as Tom suggested.
>
> OK. What about this?
> on_error {stop|ignore|other_future_option}
> where other_future_option might be compound like "file 'copy.log'" or
> "table 'copy_log'".
+1
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-03-28 01:20 ` Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-03-28 01:20 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: torikoshia <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Hi,
On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]> wrote:
>
> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov <[email protected]> wrote:
> >
> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> > > On 2024-01-18 10:10, jian he wrote:
> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> > > > wrote:
> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> > > >> > You will need a separate parameter anyway to specify the destination
> > > >> > of "log", unless "none" became an illegal table name when I wasn't
> > > >> > looking. I don't buy that one parameter that has some special values
> > > >> > while other values could be names will be a good design. Moreover,
> > > >> > what if we want to support (say) log-to-file along with log-to-table?
> > > >> > Trying to distinguish a file name from a table name without any other
> > > >> > context seems impossible.
> > > >>
> > > >> I've been thinking we can add more values to this option to log errors
> > > >> not only to the server logs but also to the error table (not sure
> > > >> details but I imagined an error table is created for each table on
> > > >> error), without an additional option for the destination name. The
> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
> > > >>
> > > >
> > > > another idea:
> > > > on_error {error|ignore|other_future_option}
> > > > if not specified then by default ERROR.
> > > > You can also specify ERROR or IGNORE for now.
> > > >
> > > > I agree, the parameter "error_action" is better than "location".
> > >
> > > I'm not sure whether error_action or on_error is better, but either way
> > > "error_action error" and "on_error error" seems a bit odd to me.
> > > I feel "stop" is better for both cases as Tom suggested.
> >
> > OK. What about this?
> > on_error {stop|ignore|other_future_option}
> > where other_future_option might be compound like "file 'copy.log'" or
> > "table 'copy_log'".
>
> +1
>
I realized that ON_ERROR syntax synoposis in the documentation is not
correct. The option doesn't require the value to be quoted and the
value can be omitted. The attached patch fixes it.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] 0001-doc-Fix-COPY-ON_ERROR-option-syntax-synopsis.patch (1.2K, ../../CAD21AoCRL-yM2NZJkfxeJGhx6r_P+dLdn0H09dE9siMKUhzoTQ@mail.gmail.com/2-0001-doc-Fix-COPY-ON_ERROR-option-syntax-synopsis.patch)
download | inline diff:
From 9a5acbff6cf1dbebc04ae221a292161d6b6cdeb0 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 28 Mar 2024 10:11:48 +0900
Subject: [PATCH] doc: Fix COPY ON_ERROR option syntax synopsis.
Oversight in b725b7eec43.
Reviewed-by:
Discussion: https://postgr.es/m/
---
doc/src/sgml/ref/copy.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 6c83e30ed0..557e344004 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,7 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
- ON_ERROR '<replaceable class="parameter">error_action</replaceable>'
+ ON_ERROR [ <replaceable class="parameter">error_action</replaceable> ]
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
--
2.39.3
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-03-28 12:38 ` torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-03-28 12:38 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-03-28 10:20, Masahiko Sawada wrote:
> Hi,
>
> On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
> wrote:
>>
>> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
>> <[email protected]> wrote:
>> >
>> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
>> > > On 2024-01-18 10:10, jian he wrote:
>> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
>> > > > wrote:
>> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> > > >> > You will need a separate parameter anyway to specify the destination
>> > > >> > of "log", unless "none" became an illegal table name when I wasn't
>> > > >> > looking. I don't buy that one parameter that has some special values
>> > > >> > while other values could be names will be a good design. Moreover,
>> > > >> > what if we want to support (say) log-to-file along with log-to-table?
>> > > >> > Trying to distinguish a file name from a table name without any other
>> > > >> > context seems impossible.
>> > > >>
>> > > >> I've been thinking we can add more values to this option to log errors
>> > > >> not only to the server logs but also to the error table (not sure
>> > > >> details but I imagined an error table is created for each table on
>> > > >> error), without an additional option for the destination name. The
>> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
>> > > >>
>> > > >
>> > > > another idea:
>> > > > on_error {error|ignore|other_future_option}
>> > > > if not specified then by default ERROR.
>> > > > You can also specify ERROR or IGNORE for now.
>> > > >
>> > > > I agree, the parameter "error_action" is better than "location".
>> > >
>> > > I'm not sure whether error_action or on_error is better, but either way
>> > > "error_action error" and "on_error error" seems a bit odd to me.
>> > > I feel "stop" is better for both cases as Tom suggested.
>> >
>> > OK. What about this?
>> > on_error {stop|ignore|other_future_option}
>> > where other_future_option might be compound like "file 'copy.log'" or
>> > "table 'copy_log'".
>>
>> +1
>>
>
> I realized that ON_ERROR syntax synoposis in the documentation is not
> correct. The option doesn't require the value to be quoted and the
> value can be omitted. The attached patch fixes it.
>
> Regards,
Thanks!
Attached patch fixes the doc, but I'm wondering perhaps it might be
better to modify the codes to prohibit abbreviation of the value.
When seeing the query which abbreviates ON_ERROR value, I feel it's not
obvious what happens compared to other options which tolerates
abbreviation of the value such as FREEZE or HEADER.
COPY t1 FROM stdin WITH (ON_ERROR);
What do you think?
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-03-28 12:54 ` Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-03-28 12:54 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]> wrote:
>
> On 2024-03-28 10:20, Masahiko Sawada wrote:
> > Hi,
> >
> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
> > wrote:
> >>
> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
> >> <[email protected]> wrote:
> >> >
> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> >> > > On 2024-01-18 10:10, jian he wrote:
> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> >> > > > wrote:
> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> >> > > >> > You will need a separate parameter anyway to specify the destination
> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
> >> > > >> > looking. I don't buy that one parameter that has some special values
> >> > > >> > while other values could be names will be a good design. Moreover,
> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
> >> > > >> > Trying to distinguish a file name from a table name without any other
> >> > > >> > context seems impossible.
> >> > > >>
> >> > > >> I've been thinking we can add more values to this option to log errors
> >> > > >> not only to the server logs but also to the error table (not sure
> >> > > >> details but I imagined an error table is created for each table on
> >> > > >> error), without an additional option for the destination name. The
> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
> >> > > >>
> >> > > >
> >> > > > another idea:
> >> > > > on_error {error|ignore|other_future_option}
> >> > > > if not specified then by default ERROR.
> >> > > > You can also specify ERROR or IGNORE for now.
> >> > > >
> >> > > > I agree, the parameter "error_action" is better than "location".
> >> > >
> >> > > I'm not sure whether error_action or on_error is better, but either way
> >> > > "error_action error" and "on_error error" seems a bit odd to me.
> >> > > I feel "stop" is better for both cases as Tom suggested.
> >> >
> >> > OK. What about this?
> >> > on_error {stop|ignore|other_future_option}
> >> > where other_future_option might be compound like "file 'copy.log'" or
> >> > "table 'copy_log'".
> >>
> >> +1
> >>
> >
> > I realized that ON_ERROR syntax synoposis in the documentation is not
> > correct. The option doesn't require the value to be quoted and the
> > value can be omitted. The attached patch fixes it.
> >
> > Regards,
>
> Thanks!
>
> Attached patch fixes the doc, but I'm wondering perhaps it might be
> better to modify the codes to prohibit abbreviation of the value.
>
> When seeing the query which abbreviates ON_ERROR value, I feel it's not
> obvious what happens compared to other options which tolerates
> abbreviation of the value such as FREEZE or HEADER.
>
> COPY t1 FROM stdin WITH (ON_ERROR);
>
> What do you think?
Indeed. Looking at options of other commands such as VACUUM and
EXPLAIN, I can see that we can omit a boolean value, but non-boolean
parameters require its value. The HEADER option is not a pure boolean
parameter but we can omit the value. It seems to be for backward
compatibility; it used to be a boolean parameter. I agree that the
above example would confuse users.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-03-29 02:54 ` torikoshia <[email protected]>
2024-04-01 02:31 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-03-29 02:54 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-03-28 21:54, Masahiko Sawada wrote:
> On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
> wrote:
>>
>> On 2024-03-28 10:20, Masahiko Sawada wrote:
>> > Hi,
>> >
>> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
>> > wrote:
>> >>
>> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
>> >> <[email protected]> wrote:
>> >> >
>> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
>> >> > > On 2024-01-18 10:10, jian he wrote:
>> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
>> >> > > > wrote:
>> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> >> > > >> > You will need a separate parameter anyway to specify the destination
>> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
>> >> > > >> > looking. I don't buy that one parameter that has some special values
>> >> > > >> > while other values could be names will be a good design. Moreover,
>> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
>> >> > > >> > Trying to distinguish a file name from a table name without any other
>> >> > > >> > context seems impossible.
>> >> > > >>
>> >> > > >> I've been thinking we can add more values to this option to log errors
>> >> > > >> not only to the server logs but also to the error table (not sure
>> >> > > >> details but I imagined an error table is created for each table on
>> >> > > >> error), without an additional option for the destination name. The
>> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
>> >> > > >>
>> >> > > >
>> >> > > > another idea:
>> >> > > > on_error {error|ignore|other_future_option}
>> >> > > > if not specified then by default ERROR.
>> >> > > > You can also specify ERROR or IGNORE for now.
>> >> > > >
>> >> > > > I agree, the parameter "error_action" is better than "location".
>> >> > >
>> >> > > I'm not sure whether error_action or on_error is better, but either way
>> >> > > "error_action error" and "on_error error" seems a bit odd to me.
>> >> > > I feel "stop" is better for both cases as Tom suggested.
>> >> >
>> >> > OK. What about this?
>> >> > on_error {stop|ignore|other_future_option}
>> >> > where other_future_option might be compound like "file 'copy.log'" or
>> >> > "table 'copy_log'".
>> >>
>> >> +1
>> >>
>> >
>> > I realized that ON_ERROR syntax synoposis in the documentation is not
>> > correct. The option doesn't require the value to be quoted and the
>> > value can be omitted. The attached patch fixes it.
>> >
>> > Regards,
>>
>> Thanks!
>>
>> Attached patch fixes the doc, but I'm wondering perhaps it might be
>> better to modify the codes to prohibit abbreviation of the value.
>>
>> When seeing the query which abbreviates ON_ERROR value, I feel it's
>> not
>> obvious what happens compared to other options which tolerates
>> abbreviation of the value such as FREEZE or HEADER.
>>
>> COPY t1 FROM stdin WITH (ON_ERROR);
>>
>> What do you think?
>
> Indeed. Looking at options of other commands such as VACUUM and
> EXPLAIN, I can see that we can omit a boolean value, but non-boolean
> parameters require its value. The HEADER option is not a pure boolean
> parameter but we can omit the value. It seems to be for backward
> compatibility; it used to be a boolean parameter. I agree that the
> above example would confuse users.
>
> Regards,
Thanks for your comment!
Attached a patch which modifies the code to prohibit omission of its
value.
I was a little unsure about adding a regression test for this, but I
have not added it since other COPY option doesn't test the omission of
its value.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
Attachments:
[text/x-diff] v1-0001-Disallow-ON_ERROR-option-without-value.patch (1.6K, ../../[email protected]/2-v1-0001-Disallow-ON_ERROR-option-without-value.patch)
download | inline diff:
From 1b4bec3c2223246ec59ffb9eb7de2f1de27315f7 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <[email protected]>
Date: Fri, 29 Mar 2024 11:36:12 +0900
Subject: [PATCH v1] Disallow ON_ERROR option without value
Currently ON_ERROR option of COPY allows to omit its value,
but the syntax synopsis in the documentation requires it.
Since it seems non-boolean parameters usually require its value
and it's not obvious what happens when value of ON_ERROR is
omitted, this patch disallows ON_ERROR without its value.
---
src/backend/commands/copy.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 28cf8b040a..2719bf28b7 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -392,7 +392,7 @@ defGetCopyHeaderChoice(DefElem *def, bool is_from)
static CopyOnErrorChoice
defGetCopyOnErrorChoice(DefElem *def, ParseState *pstate, bool is_from)
{
- char *sval;
+ char *sval = defGetString(def);
if (!is_from)
ereport(ERROR,
@@ -400,16 +400,9 @@ defGetCopyOnErrorChoice(DefElem *def, ParseState *pstate, bool is_from)
errmsg("COPY ON_ERROR cannot be used with COPY TO"),
parser_errposition(pstate, def->location)));
- /*
- * If no parameter value given, assume the default value.
- */
- if (def->arg == NULL)
- return COPY_ON_ERROR_STOP;
-
/*
* Allow "stop", or "ignore" values.
*/
- sval = defGetString(def);
if (pg_strcasecmp(sval, "stop") == 0)
return COPY_ON_ERROR_STOP;
if (pg_strcasecmp(sval, "ignore") == 0)
base-commit: 0075d78947e3800c5a807f48fd901f16db91101b
--
2.39.2
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-04-01 02:31 ` Masahiko Sawada <[email protected]>
2024-04-02 10:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-04-01 02:31 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Fri, Mar 29, 2024 at 11:54 AM torikoshia <[email protected]> wrote:
>
> On 2024-03-28 21:54, Masahiko Sawada wrote:
> > On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
> > wrote:
> >>
> >> On 2024-03-28 10:20, Masahiko Sawada wrote:
> >> > Hi,
> >> >
> >> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
> >> > wrote:
> >> >>
> >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
> >> >> <[email protected]> wrote:
> >> >> >
> >> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> >> >> > > On 2024-01-18 10:10, jian he wrote:
> >> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> >> >> > > > wrote:
> >> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> >> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> >> >> > > >> > You will need a separate parameter anyway to specify the destination
> >> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
> >> >> > > >> > looking. I don't buy that one parameter that has some special values
> >> >> > > >> > while other values could be names will be a good design. Moreover,
> >> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
> >> >> > > >> > Trying to distinguish a file name from a table name without any other
> >> >> > > >> > context seems impossible.
> >> >> > > >>
> >> >> > > >> I've been thinking we can add more values to this option to log errors
> >> >> > > >> not only to the server logs but also to the error table (not sure
> >> >> > > >> details but I imagined an error table is created for each table on
> >> >> > > >> error), without an additional option for the destination name. The
> >> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
> >> >> > > >>
> >> >> > > >
> >> >> > > > another idea:
> >> >> > > > on_error {error|ignore|other_future_option}
> >> >> > > > if not specified then by default ERROR.
> >> >> > > > You can also specify ERROR or IGNORE for now.
> >> >> > > >
> >> >> > > > I agree, the parameter "error_action" is better than "location".
> >> >> > >
> >> >> > > I'm not sure whether error_action or on_error is better, but either way
> >> >> > > "error_action error" and "on_error error" seems a bit odd to me.
> >> >> > > I feel "stop" is better for both cases as Tom suggested.
> >> >> >
> >> >> > OK. What about this?
> >> >> > on_error {stop|ignore|other_future_option}
> >> >> > where other_future_option might be compound like "file 'copy.log'" or
> >> >> > "table 'copy_log'".
> >> >>
> >> >> +1
> >> >>
> >> >
> >> > I realized that ON_ERROR syntax synoposis in the documentation is not
> >> > correct. The option doesn't require the value to be quoted and the
> >> > value can be omitted. The attached patch fixes it.
> >> >
> >> > Regards,
> >>
> >> Thanks!
> >>
> >> Attached patch fixes the doc, but I'm wondering perhaps it might be
> >> better to modify the codes to prohibit abbreviation of the value.
> >>
> >> When seeing the query which abbreviates ON_ERROR value, I feel it's
> >> not
> >> obvious what happens compared to other options which tolerates
> >> abbreviation of the value such as FREEZE or HEADER.
> >>
> >> COPY t1 FROM stdin WITH (ON_ERROR);
> >>
> >> What do you think?
> >
> > Indeed. Looking at options of other commands such as VACUUM and
> > EXPLAIN, I can see that we can omit a boolean value, but non-boolean
> > parameters require its value. The HEADER option is not a pure boolean
> > parameter but we can omit the value. It seems to be for backward
> > compatibility; it used to be a boolean parameter. I agree that the
> > above example would confuse users.
> >
> > Regards,
>
> Thanks for your comment!
>
> Attached a patch which modifies the code to prohibit omission of its
> value.
>
> I was a little unsure about adding a regression test for this, but I
> have not added it since other COPY option doesn't test the omission of
> its value.
Probably should we change the doc as well since ON_ERROR value doesn't
necessarily need to be single-quoted?
The rest looks good to me.
Alexander, what do you think about this change as you're the committer
of this feature?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-01 02:31 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-04-02 10:34 ` torikoshia <[email protected]>
2024-04-16 04:16 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-04-02 10:34 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-04-01 11:31, Masahiko Sawada wrote:
> On Fri, Mar 29, 2024 at 11:54 AM torikoshia
> <[email protected]> wrote:
>>
>> On 2024-03-28 21:54, Masahiko Sawada wrote:
>> > On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
>> > wrote:
>> >>
>> >> On 2024-03-28 10:20, Masahiko Sawada wrote:
>> >> > Hi,
>> >> >
>> >> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
>> >> > wrote:
>> >> >>
>> >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
>> >> >> <[email protected]> wrote:
>> >> >> >
>> >> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
>> >> >> > > On 2024-01-18 10:10, jian he wrote:
>> >> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
>> >> >> > > > wrote:
>> >> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> >> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> >> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> >> >> > > >> > You will need a separate parameter anyway to specify the destination
>> >> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
>> >> >> > > >> > looking. I don't buy that one parameter that has some special values
>> >> >> > > >> > while other values could be names will be a good design. Moreover,
>> >> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
>> >> >> > > >> > Trying to distinguish a file name from a table name without any other
>> >> >> > > >> > context seems impossible.
>> >> >> > > >>
>> >> >> > > >> I've been thinking we can add more values to this option to log errors
>> >> >> > > >> not only to the server logs but also to the error table (not sure
>> >> >> > > >> details but I imagined an error table is created for each table on
>> >> >> > > >> error), without an additional option for the destination name. The
>> >> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
>> >> >> > > >>
>> >> >> > > >
>> >> >> > > > another idea:
>> >> >> > > > on_error {error|ignore|other_future_option}
>> >> >> > > > if not specified then by default ERROR.
>> >> >> > > > You can also specify ERROR or IGNORE for now.
>> >> >> > > >
>> >> >> > > > I agree, the parameter "error_action" is better than "location".
>> >> >> > >
>> >> >> > > I'm not sure whether error_action or on_error is better, but either way
>> >> >> > > "error_action error" and "on_error error" seems a bit odd to me.
>> >> >> > > I feel "stop" is better for both cases as Tom suggested.
>> >> >> >
>> >> >> > OK. What about this?
>> >> >> > on_error {stop|ignore|other_future_option}
>> >> >> > where other_future_option might be compound like "file 'copy.log'" or
>> >> >> > "table 'copy_log'".
>> >> >>
>> >> >> +1
>> >> >>
>> >> >
>> >> > I realized that ON_ERROR syntax synoposis in the documentation is not
>> >> > correct. The option doesn't require the value to be quoted and the
>> >> > value can be omitted. The attached patch fixes it.
>> >> >
>> >> > Regards,
>> >>
>> >> Thanks!
>> >>
>> >> Attached patch fixes the doc, but I'm wondering perhaps it might be
>> >> better to modify the codes to prohibit abbreviation of the value.
>> >>
>> >> When seeing the query which abbreviates ON_ERROR value, I feel it's
>> >> not
>> >> obvious what happens compared to other options which tolerates
>> >> abbreviation of the value such as FREEZE or HEADER.
>> >>
>> >> COPY t1 FROM stdin WITH (ON_ERROR);
>> >>
>> >> What do you think?
>> >
>> > Indeed. Looking at options of other commands such as VACUUM and
>> > EXPLAIN, I can see that we can omit a boolean value, but non-boolean
>> > parameters require its value. The HEADER option is not a pure boolean
>> > parameter but we can omit the value. It seems to be for backward
>> > compatibility; it used to be a boolean parameter. I agree that the
>> > above example would confuse users.
>> >
>> > Regards,
>>
>> Thanks for your comment!
>>
>> Attached a patch which modifies the code to prohibit omission of its
>> value.
>>
>> I was a little unsure about adding a regression test for this, but I
>> have not added it since other COPY option doesn't test the omission of
>> its value.
>
> Probably should we change the doc as well since ON_ERROR value doesn't
> necessarily need to be single-quoted?
Agreed.
Since it seems this issue is independent from the omission of ON_ERROR
option value, attached a separate patch.
> The rest looks good to me.
>
> Alexander, what do you think about this change as you're the committer
> of this feature?
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
Attachments:
[text/x-diff] v1-0001-Disallow-ON_ERROR-option-without-value.patch (1.6K, ../../[email protected]/2-v1-0001-Disallow-ON_ERROR-option-without-value.patch)
download | inline diff:
From 1b4bec3c2223246ec59ffb9eb7de2f1de27315f7 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <[email protected]>
Date: Fri, 29 Mar 2024 11:36:12 +0900
Subject: [PATCH v1] Disallow ON_ERROR option without value
Currently ON_ERROR option of COPY allows to omit its value,
but the syntax synopsis in the documentation requires it.
Since it seems non-boolean parameters usually require its value
and it's not obvious what happens when value of ON_ERROR is
omitted, this patch disallows ON_ERROR without its value.
---
src/backend/commands/copy.c | 9 +--------
1 file changed, 1 insertion(+), 8 deletions(-)
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 28cf8b040a..2719bf28b7 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -392,7 +392,7 @@ defGetCopyHeaderChoice(DefElem *def, bool is_from)
static CopyOnErrorChoice
defGetCopyOnErrorChoice(DefElem *def, ParseState *pstate, bool is_from)
{
- char *sval;
+ char *sval = defGetString(def);
if (!is_from)
ereport(ERROR,
@@ -400,16 +400,9 @@ defGetCopyOnErrorChoice(DefElem *def, ParseState *pstate, bool is_from)
errmsg("COPY ON_ERROR cannot be used with COPY TO"),
parser_errposition(pstate, def->location)));
- /*
- * If no parameter value given, assume the default value.
- */
- if (def->arg == NULL)
- return COPY_ON_ERROR_STOP;
-
/*
* Allow "stop", or "ignore" values.
*/
- sval = defGetString(def);
if (pg_strcasecmp(sval, "stop") == 0)
return COPY_ON_ERROR_STOP;
if (pg_strcasecmp(sval, "ignore") == 0)
base-commit: 0075d78947e3800c5a807f48fd901f16db91101b
--
2.39.2
[text/x-diff] v1-0001-doc-Fix-COPY-ON_ERROR-option-syntax-synopsis.patch (1.3K, ../../[email protected]/3-v1-0001-doc-Fix-COPY-ON_ERROR-option-syntax-synopsis.patch)
download | inline diff:
From 840152c20d47220f106d5fe14af4a86cec99987e Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <[email protected]>
Date: Tue, 2 Apr 2024 19:11:01 +0900
Subject: [PATCH v1] doc: Fix COPY ON_ERROR option syntax synopsis.
Since ON_ERROR value doesn't require quotations, this patch removes them.
Oversight in b725b7eec43.
---
doc/src/sgml/ref/copy.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 33ce7c4ea6..1ce19668d8 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,7 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
- ON_ERROR '<replaceable class="parameter">error_action</replaceable>'
+ ON_ERROR <replaceable class="parameter">error_action</replaceable>
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
LOG_VERBOSITY <replaceable class="parameter">mode</replaceable>
</synopsis>
base-commit: 0075d78947e3800c5a807f48fd901f16db91101b
--
2.39.2
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-01 02:31 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-04-02 10:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-04-16 04:16 ` Masahiko Sawada <[email protected]>
2024-04-17 07:28 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Masahiko Sawada @ 2024-04-16 04:16 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Tue, Apr 2, 2024 at 7:34 PM torikoshia <[email protected]> wrote:
>
> On 2024-04-01 11:31, Masahiko Sawada wrote:
> > On Fri, Mar 29, 2024 at 11:54 AM torikoshia
> > <[email protected]> wrote:
> >>
> >> On 2024-03-28 21:54, Masahiko Sawada wrote:
> >> > On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
> >> > wrote:
> >> >>
> >> >> On 2024-03-28 10:20, Masahiko Sawada wrote:
> >> >> > Hi,
> >> >> >
> >> >> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
> >> >> > wrote:
> >> >> >>
> >> >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
> >> >> >> <[email protected]> wrote:
> >> >> >> >
> >> >> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> >> >> >> > > On 2024-01-18 10:10, jian he wrote:
> >> >> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> >> >> >> > > > wrote:
> >> >> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >> >> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> >> >> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> >> >> >> > > >> > You will need a separate parameter anyway to specify the destination
> >> >> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
> >> >> >> > > >> > looking. I don't buy that one parameter that has some special values
> >> >> >> > > >> > while other values could be names will be a good design. Moreover,
> >> >> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
> >> >> >> > > >> > Trying to distinguish a file name from a table name without any other
> >> >> >> > > >> > context seems impossible.
> >> >> >> > > >>
> >> >> >> > > >> I've been thinking we can add more values to this option to log errors
> >> >> >> > > >> not only to the server logs but also to the error table (not sure
> >> >> >> > > >> details but I imagined an error table is created for each table on
> >> >> >> > > >> error), without an additional option for the destination name. The
> >> >> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
> >> >> >> > > >>
> >> >> >> > > >
> >> >> >> > > > another idea:
> >> >> >> > > > on_error {error|ignore|other_future_option}
> >> >> >> > > > if not specified then by default ERROR.
> >> >> >> > > > You can also specify ERROR or IGNORE for now.
> >> >> >> > > >
> >> >> >> > > > I agree, the parameter "error_action" is better than "location".
> >> >> >> > >
> >> >> >> > > I'm not sure whether error_action or on_error is better, but either way
> >> >> >> > > "error_action error" and "on_error error" seems a bit odd to me.
> >> >> >> > > I feel "stop" is better for both cases as Tom suggested.
> >> >> >> >
> >> >> >> > OK. What about this?
> >> >> >> > on_error {stop|ignore|other_future_option}
> >> >> >> > where other_future_option might be compound like "file 'copy.log'" or
> >> >> >> > "table 'copy_log'".
> >> >> >>
> >> >> >> +1
> >> >> >>
> >> >> >
> >> >> > I realized that ON_ERROR syntax synoposis in the documentation is not
> >> >> > correct. The option doesn't require the value to be quoted and the
> >> >> > value can be omitted. The attached patch fixes it.
> >> >> >
> >> >> > Regards,
> >> >>
> >> >> Thanks!
> >> >>
> >> >> Attached patch fixes the doc, but I'm wondering perhaps it might be
> >> >> better to modify the codes to prohibit abbreviation of the value.
> >> >>
> >> >> When seeing the query which abbreviates ON_ERROR value, I feel it's
> >> >> not
> >> >> obvious what happens compared to other options which tolerates
> >> >> abbreviation of the value such as FREEZE or HEADER.
> >> >>
> >> >> COPY t1 FROM stdin WITH (ON_ERROR);
> >> >>
> >> >> What do you think?
> >> >
> >> > Indeed. Looking at options of other commands such as VACUUM and
> >> > EXPLAIN, I can see that we can omit a boolean value, but non-boolean
> >> > parameters require its value. The HEADER option is not a pure boolean
> >> > parameter but we can omit the value. It seems to be for backward
> >> > compatibility; it used to be a boolean parameter. I agree that the
> >> > above example would confuse users.
> >> >
> >> > Regards,
> >>
> >> Thanks for your comment!
> >>
> >> Attached a patch which modifies the code to prohibit omission of its
> >> value.
> >>
> >> I was a little unsure about adding a regression test for this, but I
> >> have not added it since other COPY option doesn't test the omission of
> >> its value.
> >
> > Probably should we change the doc as well since ON_ERROR value doesn't
> > necessarily need to be single-quoted?
>
> Agreed.
> Since it seems this issue is independent from the omission of ON_ERROR
> option value, attached a separate patch.
>
Thank you for the patches! These patches look good to me. I'll push
them, barring any objections.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-01 02:31 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-04-02 10:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-16 04:16 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2024-04-17 07:28 ` torikoshia <[email protected]>
2024-04-17 07:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-04-17 07:28 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-04-16 13:16, Masahiko Sawada wrote:
> On Tue, Apr 2, 2024 at 7:34 PM torikoshia <[email protected]>
> wrote:
>>
>> On 2024-04-01 11:31, Masahiko Sawada wrote:
>> > On Fri, Mar 29, 2024 at 11:54 AM torikoshia
>> > <[email protected]> wrote:
>> >>
>> >> On 2024-03-28 21:54, Masahiko Sawada wrote:
>> >> > On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
>> >> > wrote:
>> >> >>
>> >> >> On 2024-03-28 10:20, Masahiko Sawada wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
>> >> >> >> <[email protected]> wrote:
>> >> >> >> >
>> >> >> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
>> >> >> >> > > On 2024-01-18 10:10, jian he wrote:
>> >> >> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
>> >> >> >> > > > wrote:
>> >> >> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> >> >> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> >> >> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> >> >> >> > > >> > You will need a separate parameter anyway to specify the destination
>> >> >> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
>> >> >> >> > > >> > looking. I don't buy that one parameter that has some special values
>> >> >> >> > > >> > while other values could be names will be a good design. Moreover,
>> >> >> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
>> >> >> >> > > >> > Trying to distinguish a file name from a table name without any other
>> >> >> >> > > >> > context seems impossible.
>> >> >> >> > > >>
>> >> >> >> > > >> I've been thinking we can add more values to this option to log errors
>> >> >> >> > > >> not only to the server logs but also to the error table (not sure
>> >> >> >> > > >> details but I imagined an error table is created for each table on
>> >> >> >> > > >> error), without an additional option for the destination name. The
>> >> >> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
>> >> >> >> > > >>
>> >> >> >> > > >
>> >> >> >> > > > another idea:
>> >> >> >> > > > on_error {error|ignore|other_future_option}
>> >> >> >> > > > if not specified then by default ERROR.
>> >> >> >> > > > You can also specify ERROR or IGNORE for now.
>> >> >> >> > > >
>> >> >> >> > > > I agree, the parameter "error_action" is better than "location".
>> >> >> >> > >
>> >> >> >> > > I'm not sure whether error_action or on_error is better, but either way
>> >> >> >> > > "error_action error" and "on_error error" seems a bit odd to me.
>> >> >> >> > > I feel "stop" is better for both cases as Tom suggested.
>> >> >> >> >
>> >> >> >> > OK. What about this?
>> >> >> >> > on_error {stop|ignore|other_future_option}
>> >> >> >> > where other_future_option might be compound like "file 'copy.log'" or
>> >> >> >> > "table 'copy_log'".
>> >> >> >>
>> >> >> >> +1
>> >> >> >>
>> >> >> >
>> >> >> > I realized that ON_ERROR syntax synoposis in the documentation is not
>> >> >> > correct. The option doesn't require the value to be quoted and the
>> >> >> > value can be omitted. The attached patch fixes it.
>> >> >> >
>> >> >> > Regards,
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >> Attached patch fixes the doc, but I'm wondering perhaps it might be
>> >> >> better to modify the codes to prohibit abbreviation of the value.
>> >> >>
>> >> >> When seeing the query which abbreviates ON_ERROR value, I feel it's
>> >> >> not
>> >> >> obvious what happens compared to other options which tolerates
>> >> >> abbreviation of the value such as FREEZE or HEADER.
>> >> >>
>> >> >> COPY t1 FROM stdin WITH (ON_ERROR);
>> >> >>
>> >> >> What do you think?
>> >> >
>> >> > Indeed. Looking at options of other commands such as VACUUM and
>> >> > EXPLAIN, I can see that we can omit a boolean value, but non-boolean
>> >> > parameters require its value. The HEADER option is not a pure boolean
>> >> > parameter but we can omit the value. It seems to be for backward
>> >> > compatibility; it used to be a boolean parameter. I agree that the
>> >> > above example would confuse users.
>> >> >
>> >> > Regards,
>> >>
>> >> Thanks for your comment!
>> >>
>> >> Attached a patch which modifies the code to prohibit omission of its
>> >> value.
>> >>
>> >> I was a little unsure about adding a regression test for this, but I
>> >> have not added it since other COPY option doesn't test the omission of
>> >> its value.
>> >
>> > Probably should we change the doc as well since ON_ERROR value doesn't
>> > necessarily need to be single-quoted?
>>
>> Agreed.
>> Since it seems this issue is independent from the omission of ON_ERROR
>> option value, attached a separate patch.
>>
>
> Thank you for the patches! These patches look good to me. I'll push
> them, barring any objections.
>
> Regards,
Thanks for your review and apply!
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 08:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-03-28 12:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-01 02:31 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-04-02 10:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-04-16 04:16 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-04-17 07:28 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-04-17 07:30 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Masahiko Sawada @ 2024-04-17 07:30 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; jian he <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On Wed, Apr 17, 2024 at 4:28 PM torikoshia <[email protected]> wrote:
>
> On 2024-04-16 13:16, Masahiko Sawada wrote:
> > On Tue, Apr 2, 2024 at 7:34 PM torikoshia <[email protected]>
> > wrote:
> >>
> >> On 2024-04-01 11:31, Masahiko Sawada wrote:
> >> > On Fri, Mar 29, 2024 at 11:54 AM torikoshia
> >> > <[email protected]> wrote:
> >> >>
> >> >> On 2024-03-28 21:54, Masahiko Sawada wrote:
> >> >> > On Thu, Mar 28, 2024 at 9:38 PM torikoshia <[email protected]>
> >> >> > wrote:
> >> >> >>
> >> >> >> On 2024-03-28 10:20, Masahiko Sawada wrote:
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada <[email protected]>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov
> >> >> >> >> <[email protected]> wrote:
> >> >> >> >> >
> >> >> >> >> > On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]> wrote:
> >> >> >> >> > > On 2024-01-18 10:10, jian he wrote:
> >> >> >> >> > > > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
> >> >> >> >> > > > wrote:
> >> >> >> >> > > >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
> >> >> >> >> > > >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
> >> >> >> >> > > >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
> >> >> >> >> > > >> > You will need a separate parameter anyway to specify the destination
> >> >> >> >> > > >> > of "log", unless "none" became an illegal table name when I wasn't
> >> >> >> >> > > >> > looking. I don't buy that one parameter that has some special values
> >> >> >> >> > > >> > while other values could be names will be a good design. Moreover,
> >> >> >> >> > > >> > what if we want to support (say) log-to-file along with log-to-table?
> >> >> >> >> > > >> > Trying to distinguish a file name from a table name without any other
> >> >> >> >> > > >> > context seems impossible.
> >> >> >> >> > > >>
> >> >> >> >> > > >> I've been thinking we can add more values to this option to log errors
> >> >> >> >> > > >> not only to the server logs but also to the error table (not sure
> >> >> >> >> > > >> details but I imagined an error table is created for each table on
> >> >> >> >> > > >> error), without an additional option for the destination name. The
> >> >> >> >> > > >> values would be like error_action {error|ignore|save-logs|save-table}.
> >> >> >> >> > > >>
> >> >> >> >> > > >
> >> >> >> >> > > > another idea:
> >> >> >> >> > > > on_error {error|ignore|other_future_option}
> >> >> >> >> > > > if not specified then by default ERROR.
> >> >> >> >> > > > You can also specify ERROR or IGNORE for now.
> >> >> >> >> > > >
> >> >> >> >> > > > I agree, the parameter "error_action" is better than "location".
> >> >> >> >> > >
> >> >> >> >> > > I'm not sure whether error_action or on_error is better, but either way
> >> >> >> >> > > "error_action error" and "on_error error" seems a bit odd to me.
> >> >> >> >> > > I feel "stop" is better for both cases as Tom suggested.
> >> >> >> >> >
> >> >> >> >> > OK. What about this?
> >> >> >> >> > on_error {stop|ignore|other_future_option}
> >> >> >> >> > where other_future_option might be compound like "file 'copy.log'" or
> >> >> >> >> > "table 'copy_log'".
> >> >> >> >>
> >> >> >> >> +1
> >> >> >> >>
> >> >> >> >
> >> >> >> > I realized that ON_ERROR syntax synoposis in the documentation is not
> >> >> >> > correct. The option doesn't require the value to be quoted and the
> >> >> >> > value can be omitted. The attached patch fixes it.
> >> >> >> >
> >> >> >> > Regards,
> >> >> >>
> >> >> >> Thanks!
> >> >> >>
> >> >> >> Attached patch fixes the doc, but I'm wondering perhaps it might be
> >> >> >> better to modify the codes to prohibit abbreviation of the value.
> >> >> >>
> >> >> >> When seeing the query which abbreviates ON_ERROR value, I feel it's
> >> >> >> not
> >> >> >> obvious what happens compared to other options which tolerates
> >> >> >> abbreviation of the value such as FREEZE or HEADER.
> >> >> >>
> >> >> >> COPY t1 FROM stdin WITH (ON_ERROR);
> >> >> >>
> >> >> >> What do you think?
> >> >> >
> >> >> > Indeed. Looking at options of other commands such as VACUUM and
> >> >> > EXPLAIN, I can see that we can omit a boolean value, but non-boolean
> >> >> > parameters require its value. The HEADER option is not a pure boolean
> >> >> > parameter but we can omit the value. It seems to be for backward
> >> >> > compatibility; it used to be a boolean parameter. I agree that the
> >> >> > above example would confuse users.
> >> >> >
> >> >> > Regards,
> >> >>
> >> >> Thanks for your comment!
> >> >>
> >> >> Attached a patch which modifies the code to prohibit omission of its
> >> >> value.
> >> >>
> >> >> I was a little unsure about adding a regression test for this, but I
> >> >> have not added it since other COPY option doesn't test the omission of
> >> >> its value.
> >> >
> >> > Probably should we change the doc as well since ON_ERROR value doesn't
> >> > necessarily need to be single-quoted?
> >>
> >> Agreed.
> >> Since it seems this issue is independent from the omission of ON_ERROR
> >> option value, attached a separate patch.
> >>
> >
> > Thank you for the patches! These patches look good to me. I'll push
> > them, barring any objections.
> >
> > Regards,
>
> Thanks for your review and apply!
Thank you for the patches!
Pushed: a6d0fa5ef8 and f6f8ac8e75.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-18 12:09 ` torikoshia <[email protected]>
2024-01-18 14:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-01-18 12:09 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: jian he <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-01-18 16:59, Alexander Korotkov wrote:
> On Thu, Jan 18, 2024 at 4:16 AM torikoshia <[email protected]>
> wrote:
>> On 2024-01-18 10:10, jian he wrote:
>> > On Thu, Jan 18, 2024 at 8:57 AM Masahiko Sawada <[email protected]>
>> > wrote:
>> >> On Thu, Jan 18, 2024 at 6:38 AM Tom Lane <[email protected]> wrote:
>> >> > Kyotaro-san's suggestion isn't bad, though I might shorten it to
>> >> > error_action {error|ignore|log} (or perhaps "stop" instead of "error")?
>> >> > You will need a separate parameter anyway to specify the destination
>> >> > of "log", unless "none" became an illegal table name when I wasn't
>> >> > looking. I don't buy that one parameter that has some special values
>> >> > while other values could be names will be a good design. Moreover,
>> >> > what if we want to support (say) log-to-file along with log-to-table?
>> >> > Trying to distinguish a file name from a table name without any other
>> >> > context seems impossible.
>> >>
>> >> I've been thinking we can add more values to this option to log errors
>> >> not only to the server logs but also to the error table (not sure
>> >> details but I imagined an error table is created for each table on
>> >> error), without an additional option for the destination name. The
>> >> values would be like error_action {error|ignore|save-logs|save-table}.
>> >>
>> >
>> > another idea:
>> > on_error {error|ignore|other_future_option}
>> > if not specified then by default ERROR.
>> > You can also specify ERROR or IGNORE for now.
>> >
>> > I agree, the parameter "error_action" is better than "location".
>>
>> I'm not sure whether error_action or on_error is better, but either
>> way
>> "error_action error" and "on_error error" seems a bit odd to me.
>> I feel "stop" is better for both cases as Tom suggested.
>
> OK. What about this?
> on_error {stop|ignore|other_future_option}
> where other_future_option might be compound like "file 'copy.log'" or
> "table 'copy_log'".
Thanks, also +1 from me.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 12:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-18 14:59 ` jian he <[email protected]>
2024-01-19 12:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: jian he @ 2024-01-18 14:59 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Hi.
patch refactored based on "on_error {stop|ignore}"
doc changes:
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,7 +43,7 @@ COPY { <replaceable
class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable
class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable
class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable
class="parameter">column_name</replaceable> [, ...] ) | * }
- SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
+ ON_ERROR '<replaceable class="parameter">error_action</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -375,20 +375,20 @@ COPY { <replaceable
class="parameter">table_name</replaceable> [ ( <replaceable
</varlistentry>
<varlistentry>
- <term><literal>SAVE_ERROR_TO</literal></term>
+ <term><literal>ON_ERROR</literal></term>
<listitem>
<para>
- Specifies to save error information to <replaceable class="parameter">
- location</replaceable> when there is malformed data in the input.
- Currently, only <literal>error</literal> (default) and
<literal>none</literal>
+ Specifies which <replaceable class="parameter">
+ error_action</replaceable> to perform when there is malformed
data in the input.
+ Currently, only <literal>stop</literal> (default) and
<literal>ignore</literal>
values are supported.
- If the <literal>error</literal> value is specified,
+ If the <literal>stop</literal> value is specified,
<command>COPY</command> stops operation at the first error.
- If the <literal>none</literal> value is specified,
+ If the <literal>ignore</literal> value is specified,
<command>COPY</command> skips malformed data and continues copying data.
The option is allowed only in <command>COPY FROM</command>.
- The <literal>none</literal> value is allowed only when
- not using <literal>binary</literal> format.
+ Only <literal>stop</literal> value is allowed only when
+ using <literal>binary</literal> format.
</para>
Attachments:
[text/x-patch] copy_on_error.diff (17.8K, ../../CACJufxGJjj94XTWsD9eJaiE99CPRBkO1frvJWTJ-6NDpa+ksLg@mail.gmail.com/2-copy_on_error.diff)
download | inline diff:
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 85881ca0..c30baec1 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -43,7 +43,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
FORCE_QUOTE { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NOT_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
FORCE_NULL { ( <replaceable class="parameter">column_name</replaceable> [, ...] ) | * }
- SAVE_ERROR_TO '<replaceable class="parameter">location</replaceable>'
+ ON_ERROR '<replaceable class="parameter">error_action</replaceable>'
ENCODING '<replaceable class="parameter">encoding_name</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -375,20 +375,20 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
</varlistentry>
<varlistentry>
- <term><literal>SAVE_ERROR_TO</literal></term>
+ <term><literal>ON_ERROR</literal></term>
<listitem>
<para>
- Specifies to save error information to <replaceable class="parameter">
- location</replaceable> when there is malformed data in the input.
- Currently, only <literal>error</literal> (default) and <literal>none</literal>
+ Specifies which <replaceable class="parameter">
+ error_action</replaceable> to perform when there is malformed data in the input.
+ Currently, only <literal>stop</literal> (default) and <literal>ignore</literal>
values are supported.
- If the <literal>error</literal> value is specified,
+ If the <literal>stop</literal> value is specified,
<command>COPY</command> stops operation at the first error.
- If the <literal>none</literal> value is specified,
+ If the <literal>ignore</literal> value is specified,
<command>COPY</command> skips malformed data and continues copying data.
The option is allowed only in <command>COPY FROM</command>.
- The <literal>none</literal> value is allowed only when
- not using <literal>binary</literal> format.
+ Only <literal>stop</literal> value is allowed only when
+ using <literal>binary</literal> format.
</para>
</listitem>
</varlistentry>
@@ -577,7 +577,7 @@ COPY <replaceable class="parameter">count</replaceable>
<para>
<command>COPY</command> stops operation at the first error when
- <literal>SAVE_ERROR_TO</literal> is not specified. This
+ <literal>ON_ERROR</literal> is not specified. This
should not lead to problems in the event of a <command>COPY
TO</command>, but the target table will already have received
earlier rows in a <command>COPY FROM</command>. These rows will not
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index c36d7f1d..cc0786c6 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -395,39 +395,39 @@ defGetCopyHeaderChoice(DefElem *def, bool is_from)
}
/*
- * Extract a CopySaveErrorToChoice value from a DefElem.
+ * Extract a CopyOnErrorChoice value from a DefElem.
*/
-static CopySaveErrorToChoice
-defGetCopySaveErrorToChoice(DefElem *def, ParseState *pstate, bool is_from)
+static CopyOnErrorChoice
+defGetCopyOnErrorChoice(DefElem *def, ParseState *pstate, bool is_from)
{
char *sval;
if (!is_from)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("COPY SAVE_ERROR_TO cannot be used with COPY TO"),
+ errmsg("COPY ON_ERROR cannot be used with COPY TO"),
parser_errposition(pstate, def->location)));
/*
* If no parameter value given, assume the default value.
*/
if (def->arg == NULL)
- return COPY_SAVE_ERROR_TO_ERROR;
+ return COPY_ON_ERROR_STOP;
/*
- * Allow "error", or "none" values.
+ * Allow "stop", or "ignore" values.
*/
sval = defGetString(def);
- if (pg_strcasecmp(sval, "error") == 0)
- return COPY_SAVE_ERROR_TO_ERROR;
- if (pg_strcasecmp(sval, "none") == 0)
- return COPY_SAVE_ERROR_TO_NONE;
+ if (pg_strcasecmp(sval, "stop") == 0)
+ return COPY_ON_ERROR_STOP;
+ if (pg_strcasecmp(sval, "ignore") == 0)
+ return COPY_ON_ERROR_IGNORE;
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("COPY save_error_to \"%s\" not recognized", sval),
+ errmsg("COPY ON_ERROR \"%s\" not recognized", sval),
parser_errposition(pstate, def->location)));
- return COPY_SAVE_ERROR_TO_ERROR; /* keep compiler quiet */
+ return COPY_ON_ERROR_STOP; /* keep compiler quiet */
}
/*
@@ -455,7 +455,7 @@ ProcessCopyOptions(ParseState *pstate,
bool format_specified = false;
bool freeze_specified = false;
bool header_specified = false;
- bool save_error_to_specified = false;
+ bool on_error_specified = false;
ListCell *option;
/* Support external use for option sanity checking */
@@ -608,12 +608,12 @@ ProcessCopyOptions(ParseState *pstate,
defel->defname),
parser_errposition(pstate, defel->location)));
}
- else if (strcmp(defel->defname, "save_error_to") == 0)
+ else if (strcmp(defel->defname, "on_error") == 0)
{
- if (save_error_to_specified)
+ if (on_error_specified)
errorConflictingDefElem(defel, pstate);
- save_error_to_specified = true;
- opts_out->save_error_to = defGetCopySaveErrorToChoice(defel, pstate, is_from);
+ on_error_specified = true;
+ opts_out->on_error = defGetCopyOnErrorChoice(defel, pstate, is_from);
}
else
ereport(ERROR,
@@ -642,10 +642,10 @@ ProcessCopyOptions(ParseState *pstate,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DEFAULT in BINARY mode")));
- if (opts_out->binary && opts_out->save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("cannot specify SAVE_ERROR_TO in BINARY mode")));
+ errmsg("only ON_ERROR STOP is allowed in BINARY mode")));
/* Set defaults for omitted options */
if (!opts_out->delim)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 50e245d5..c956cfa4 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -657,7 +657,7 @@ CopyFrom(CopyFromState cstate)
Assert(cstate->rel);
Assert(list_length(cstate->range_table) == 1);
- if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ if (cstate->opts.on_error != COPY_ON_ERROR_STOP)
Assert(cstate->escontext);
/*
@@ -996,14 +996,14 @@ CopyFrom(CopyFromState cstate)
if (!NextCopyFrom(cstate, econtext, myslot->tts_values, myslot->tts_isnull))
break;
- if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ if (cstate->opts.on_error != COPY_ON_ERROR_STOP &&
cstate->escontext->error_occurred)
{
/*
- * Soft error occured, skip this tuple and save error information
- * according to SAVE_ERROR_TO.
+ * Soft error occured, skip this tuple and deal with error information
+ * according to ON_ERROR.
*/
- if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ if (cstate->opts.on_error == COPY_ON_ERROR_IGNORE)
/*
* Just make ErrorSaveContext ready for the next NextCopyFrom.
@@ -1307,7 +1307,7 @@ CopyFrom(CopyFromState cstate)
/* Done, clean up */
error_context_stack = errcallback.previous;
- if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR &&
+ if (cstate->opts.on_error != COPY_ON_ERROR_STOP &&
cstate->num_errors > 0)
ereport(NOTICE,
errmsg_plural("%llu row was skipped due to data type incompatibility",
@@ -1450,18 +1450,18 @@ BeginCopyFrom(ParseState *pstate,
}
}
- /* Set up soft error handler for SAVE_ERROR_TO */
- if (cstate->opts.save_error_to != COPY_SAVE_ERROR_TO_ERROR)
+ /* Set up soft error handler for ON_ERROR */
+ if (cstate->opts.on_error != COPY_ON_ERROR_STOP)
{
cstate->escontext = makeNode(ErrorSaveContext);
cstate->escontext->type = T_ErrorSaveContext;
cstate->escontext->error_occurred = false;
/*
- * Currently we only support COPY_SAVE_ERROR_TO_NONE. We'll add other
+ * Currently we only support COPY_ON_ERROR_IGNORE. We'll add other
* options later
*/
- if (cstate->opts.save_error_to == COPY_SAVE_ERROR_TO_NONE)
+ if (cstate->opts.on_error == COPY_ON_ERROR_IGNORE)
cstate->escontext->details_wanted = false;
}
else
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 7207eb26..36214aab 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -956,7 +956,7 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}
- /* If SAVE_ERROR_TO is specified, skip rows with soft errors */
+ /* If ON_ERROR is specified with IGNORE, skip rows with soft errors */
else if (!InputFunctionCallSafe(&in_functions[m],
string,
typioparams[m],
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 6bfdb5f0..ada711d0 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -2899,15 +2899,15 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("FORMAT", "FREEZE", "DELIMITER", "NULL",
"HEADER", "QUOTE", "ESCAPE", "FORCE_QUOTE",
"FORCE_NOT_NULL", "FORCE_NULL", "ENCODING", "DEFAULT",
- "SAVE_ERROR_TO");
+ "ON_ERROR");
/* Complete COPY <sth> FROM|TO filename WITH (FORMAT */
else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "FORMAT"))
COMPLETE_WITH("binary", "csv", "text");
- /* Complete COPY <sth> FROM filename WITH (SAVE_ERROR_TO */
- else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "SAVE_ERROR_TO"))
- COMPLETE_WITH("error", "none");
+ /* Complete COPY <sth> FROM filename WITH (ON_ERROR */
+ else if (Matches("COPY|\\copy", MatchAny, "FROM|TO", MatchAny, "WITH", "(", "ON_ERROR"))
+ COMPLETE_WITH("stop", "ignore");
/* Complete COPY <sth> FROM <sth> WITH (<options>) */
else if (Matches("COPY|\\copy", MatchAny, "FROM", MatchAny, "WITH", MatchAny))
diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 8972c618..78af1b0e 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -34,11 +34,11 @@ typedef enum CopyHeaderChoice
* Represents where to save input processing errors. More values to be added
* in the future.
*/
-typedef enum CopySaveErrorToChoice
+typedef enum CopyOnErrorChoice
{
- COPY_SAVE_ERROR_TO_ERROR = 0, /* immediately throw errors */
- COPY_SAVE_ERROR_TO_NONE, /* ignore errors */
-} CopySaveErrorToChoice;
+ COPY_ON_ERROR_STOP = 0, /* immediately throw errors, default */
+ COPY_ON_ERROR_IGNORE, /* ignore errors */
+} CopyOnErrorChoice;
/*
* A struct to hold COPY options, in a parsed form. All of these are related
@@ -72,7 +72,7 @@ typedef struct CopyFormatOptions
bool force_null_all; /* FORCE_NULL *? */
bool *force_null_flags; /* per-column CSV FN flags */
bool convert_selectively; /* do selective binary conversion? */
- CopySaveErrorToChoice save_error_to; /* where to save error information */
+ CopyOnErrorChoice on_error; /* what to do when error happened */
List *convert_select; /* list of column names (can be NIL) */
} CopyFormatOptions;
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 42cbcb2e..d982ae4f 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -77,21 +77,21 @@ COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
ERROR: conflicting or redundant options
LINE 1: COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii...
^
-COPY x from stdin (save_error_to none,save_error_to none);
+COPY x from stdin (ON_ERROR ignore, ON_ERROR ignore);
ERROR: conflicting or redundant options
-LINE 1: COPY x from stdin (save_error_to none,save_error_to none);
- ^
+LINE 1: COPY x from stdin (ON_ERROR ignore, ON_ERROR ignore);
+ ^
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
ERROR: cannot specify DELIMITER in BINARY mode
COPY x to stdin (format BINARY, null 'x');
ERROR: cannot specify NULL in BINARY mode
-COPY x from stdin (format BINARY, save_error_to none);
-ERROR: cannot specify SAVE_ERROR_TO in BINARY mode
-COPY x to stdin (save_error_to none);
-ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
-LINE 1: COPY x to stdin (save_error_to none);
- ^
+COPY x from stdin (format BINARY, ON_ERROR ignore);
+ERROR: only ON_ERROR STOP is allowed in BINARY mode
+COPY x from stdin (ON_ERROR unsupported);
+ERROR: COPY ON_ERROR "unsupported" not recognized
+LINE 1: COPY x from stdin (ON_ERROR unsupported);
+ ^
COPY x to stdin (format TEXT, force_quote(a));
ERROR: COPY FORCE_QUOTE requires CSV mode
COPY x from stdin (format CSV, force_quote(a));
@@ -104,9 +104,9 @@ COPY x to stdout (format TEXT, force_null(a));
ERROR: COPY FORCE_NULL requires CSV mode
COPY x to stdin (format CSV, force_null(a));
ERROR: COPY FORCE_NULL cannot be used with COPY TO
-COPY x to stdin (format BINARY, save_error_to unsupported);
-ERROR: COPY SAVE_ERROR_TO cannot be used with COPY TO
-LINE 1: COPY x to stdin (format BINARY, save_error_to unsupported);
+COPY x to stdin (format BINARY, ON_ERROR unsupported);
+ERROR: COPY ON_ERROR cannot be used with COPY TO
+LINE 1: COPY x to stdin (format BINARY, ON_ERROR unsupported);
^
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -724,12 +724,12 @@ SELECT * FROM instead_of_insert_tbl;
(2 rows)
COMMIT;
--- tests for SAVE_ERROR_TO option
+-- tests for ON_ERROR option
CREATE TABLE check_ign_err (n int, m int[], k int);
-COPY check_ign_err FROM STDIN WITH (save_error_to error);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR stop);
ERROR: invalid input syntax for type integer: "a"
CONTEXT: COPY check_ign_err, line 2, column n: "a"
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
NOTICE: 4 rows were skipped due to data type incompatibility
SELECT * FROM check_ign_err;
n | m | k
@@ -740,15 +740,15 @@ SELECT * FROM check_ign_err;
-- test datatype error that can't be handled as soft: should fail
CREATE TABLE hard_err(foo widget);
-COPY hard_err FROM STDIN WITH (save_error_to none);
+COPY hard_err FROM STDIN WITH (ON_ERROR ignore);
ERROR: invalid input syntax for type widget: "1"
CONTEXT: COPY hard_err, line 1, column foo: "1"
-- test missing data: should fail
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
ERROR: missing data for column "k"
CONTEXT: COPY check_ign_err, line 1: "1 {1}"
-- test extra data: should fail
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
ERROR: extra data after last expected column
CONTEXT: COPY check_ign_err, line 1: "1 {1} 3 abc"
-- clean up
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index c48d5563..73b2e688 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -66,20 +66,20 @@ COPY x from stdin (force_not_null (a), force_not_null (b));
COPY x from stdin (force_null (a), force_null (b));
COPY x from stdin (convert_selectively (a), convert_selectively (b));
COPY x from stdin (encoding 'sql_ascii', encoding 'sql_ascii');
-COPY x from stdin (save_error_to none,save_error_to none);
+COPY x from stdin (ON_ERROR ignore, ON_ERROR ignore);
-- incorrect options
COPY x to stdin (format BINARY, delimiter ',');
COPY x to stdin (format BINARY, null 'x');
-COPY x from stdin (format BINARY, save_error_to none);
-COPY x to stdin (save_error_to none);
+COPY x from stdin (format BINARY, ON_ERROR ignore);
+COPY x from stdin (ON_ERROR unsupported);
COPY x to stdin (format TEXT, force_quote(a));
COPY x from stdin (format CSV, force_quote(a));
COPY x to stdout (format TEXT, force_not_null(a));
COPY x to stdin (format CSV, force_not_null(a));
COPY x to stdout (format TEXT, force_null(a));
COPY x to stdin (format CSV, force_null(a));
-COPY x to stdin (format BINARY, save_error_to unsupported);
+COPY x to stdin (format BINARY, ON_ERROR unsupported);
-- too many columns in column list: should fail
COPY x (a, b, c, d, e, d, c) from stdin;
@@ -498,9 +498,9 @@ test1
SELECT * FROM instead_of_insert_tbl;
COMMIT;
--- tests for SAVE_ERROR_TO option
+-- tests for ON_ERROR option
CREATE TABLE check_ign_err (n int, m int[], k int);
-COPY check_ign_err FROM STDIN WITH (save_error_to error);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR stop);
1 {1} 1
a {2} 2
3 {3} 3333333333
@@ -508,7 +508,7 @@ a {2} 2
5 {5} 5
\.
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
1 {1} 1
a {2} 2
3 {3} 3333333333
@@ -520,17 +520,17 @@ SELECT * FROM check_ign_err;
-- test datatype error that can't be handled as soft: should fail
CREATE TABLE hard_err(foo widget);
-COPY hard_err FROM STDIN WITH (save_error_to none);
+COPY hard_err FROM STDIN WITH (ON_ERROR ignore);
1
\.
-- test missing data: should fail
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
1 {1}
\.
-- test extra data: should fail
-COPY check_ign_err FROM STDIN WITH (save_error_to none);
+COPY check_ign_err FROM STDIN WITH (ON_ERROR ignore);
1 {1} 3 abc
\.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 29fd1cae..456461f8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -478,6 +478,7 @@ CopyHeaderChoice
CopyInsertMethod
CopyMultiInsertBuffer
CopyMultiInsertInfo
+CopyOnErrorChoice
CopySource
CopyStmt
CopyToState
@@ -4041,4 +4042,3 @@ manifest_writer
rfile
ws_options
ws_file_info
-CopySaveErrorToChoice
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 12:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 14:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2024-01-19 12:37 ` torikoshia <[email protected]>
2024-01-19 13:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: torikoshia @ 2024-01-19 12:37 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-01-18 23:59, jian he wrote:
> Hi.
> patch refactored based on "on_error {stop|ignore}"
> doc changes:
>
> --- a/doc/src/sgml/ref/copy.sgml
> +++ b/doc/src/sgml/ref/copy.sgml
> @@ -43,7 +43,7 @@ COPY { <replaceable
> class="parameter">table_name</replaceable> [ ( <replaceable
> FORCE_QUOTE { ( <replaceable
> class="parameter">column_name</replaceable> [, ...] ) | * }
> FORCE_NOT_NULL { ( <replaceable
> class="parameter">column_name</replaceable> [, ...] ) | * }
> FORCE_NULL { ( <replaceable
> class="parameter">column_name</replaceable> [, ...] ) | * }
> - SAVE_ERROR_TO '<replaceable
> class="parameter">location</replaceable>'
> + ON_ERROR '<replaceable
> class="parameter">error_action</replaceable>'
> ENCODING '<replaceable
> class="parameter">encoding_name</replaceable>'
> </synopsis>
> </refsynopsisdiv>
> @@ -375,20 +375,20 @@ COPY { <replaceable
> class="parameter">table_name</replaceable> [ ( <replaceable
> </varlistentry>
>
> <varlistentry>
> - <term><literal>SAVE_ERROR_TO</literal></term>
> + <term><literal>ON_ERROR</literal></term>
> <listitem>
> <para>
> - Specifies to save error information to <replaceable
> class="parameter">
> - location</replaceable> when there is malformed data in the
> input.
> - Currently, only <literal>error</literal> (default) and
> <literal>none</literal>
> + Specifies which <replaceable class="parameter">
> + error_action</replaceable> to perform when there is malformed
> data in the input.
> + Currently, only <literal>stop</literal> (default) and
> <literal>ignore</literal>
> values are supported.
> - If the <literal>error</literal> value is specified,
> + If the <literal>stop</literal> value is specified,
> <command>COPY</command> stops operation at the first error.
> - If the <literal>none</literal> value is specified,
> + If the <literal>ignore</literal> value is specified,
> <command>COPY</command> skips malformed data and continues
> copying data.
> The option is allowed only in <command>COPY FROM</command>.
> - The <literal>none</literal> value is allowed only when
> - not using <literal>binary</literal> format.
> + Only <literal>stop</literal> value is allowed only when
> + using <literal>binary</literal> format.
> </para>
Thanks for making the patch!
Here are some comments:
> - The <literal>none</literal> value is allowed only when
> - not using <literal>binary</literal> format.
> + Only <literal>stop</literal> value is allowed only when
> + using <literal>binary</literal> format.
The second 'only' may be unnecessary.
> - /* If SAVE_ERROR_TO is specified, skip rows
> with soft errors */
> + /* If ON_ERROR is specified with IGNORE, skip
> rows with soft errors */
This is correct now, but considering future works which add other
options like "file 'copy.log'" and
"table 'copy_log'", it may be better not to limit the case to 'IGNORE'.
How about something like this?
If ON_ERROR is specified and the value is not STOP, skip rows with
soft errors
> -COPY x from stdin (format BINARY, save_error_to none);
> -COPY x to stdin (save_error_to none);
> +COPY x from stdin (format BINARY, ON_ERROR ignore);
> +COPY x from stdin (ON_ERROR unsupported);
> COPY x to stdin (format TEXT, force_quote(a));
> COPY x from stdin (format CSV, force_quote(a));
In the existing test for copy2.sql, the COPY options are written in
lower case(e.g. 'format') and option value(e.g. 'BINARY') are written in
upper case.
It would be more consistent to align them.
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 12:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 14:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-19 12:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-19 13:27 ` Alexander Korotkov <[email protected]>
2024-01-19 14:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-19 13:27 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: jian he <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Hi!
On Fri, Jan 19, 2024 at 2:37 PM torikoshia <[email protected]> wrote:
> Thanks for making the patch!
The patch is pushed! The proposed changes are incorporated excluding this.
> > - /* If SAVE_ERROR_TO is specified, skip rows
> > with soft errors */
> > + /* If ON_ERROR is specified with IGNORE, skip
> > rows with soft errors */
>
> This is correct now, but considering future works which add other
> options like "file 'copy.log'" and
> "table 'copy_log'", it may be better not to limit the case to 'IGNORE'.
> How about something like this?
>
> If ON_ERROR is specified and the value is not STOP, skip rows with
> soft errors
I think when we have more options, then we wouldn't just skip rows
with soft errors but rather save them. So, I left this comment as is
for now.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-17 07:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2024-01-18 00:56 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-18 02:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 07:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-18 12:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-18 14:59 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-19 12:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-19 13:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
@ 2024-01-19 14:26 ` torikoshia <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: torikoshia @ 2024-01-19 14:26 UTC (permalink / raw)
To: Alexander Korotkov <[email protected]>; +Cc: jian he <[email protected]>; Masahiko Sawada <[email protected]>; Tom Lane <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
On 2024-01-19 22:27, Alexander Korotkov wrote:
> Hi!
>
> On Fri, Jan 19, 2024 at 2:37 PM torikoshia <[email protected]>
> wrote:
>> Thanks for making the patch!
>
> The patch is pushed! The proposed changes are incorporated excluding
> this.
>
>> > - /* If SAVE_ERROR_TO is specified, skip rows
>> > with soft errors */
>> > + /* If ON_ERROR is specified with IGNORE, skip
>> > rows with soft errors */
>>
>> This is correct now, but considering future works which add other
>> options like "file 'copy.log'" and
>> "table 'copy_log'", it may be better not to limit the case to
>> 'IGNORE'.
>> How about something like this?
>>
>> If ON_ERROR is specified and the value is not STOP, skip rows with
>> soft errors
>
> I think when we have more options, then we wouldn't just skip rows
> with soft errors but rather save them. So, I left this comment as is
> for now.
Agreed.
Thanks for the notification!
>
> ------
> Regards,
> Alexander Korotkov
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-18 05:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-12-18 07:41 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-19 01:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-20 12:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-04 16:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) vignesh C <[email protected]>
2024-01-05 08:37 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-06 00:50 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-09 14:36 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-11 03:13 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-12 02:58 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-13 14:19 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2024-01-14 01:30 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2024-01-16 15:08 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
@ 2024-01-17 21:01 ` Alexander Korotkov <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Alexander Korotkov @ 2024-01-17 21:01 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; jian he <[email protected]>; vignesh C <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Wed, Jan 17, 2024 at 7:38 AM torikoshia <[email protected]> wrote:
>
> Hi,
>
> Thanks for applying!
>
> > + errmsg_plural("%zd row were skipped due
> > to data type incompatibility",
>
> Sorry, I just noticed it, but 'were' should be 'was' here?
Sure, the fix is pushed.
------
Regards,
Alexander Korotkov
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
@ 2023-12-19 00:28 ` Masahiko Sawada <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: Masahiko Sawada @ 2023-12-19 00:28 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On Mon, Dec 18, 2023 at 9:16 AM jian he <[email protected]> wrote:
>
> On Fri, Dec 15, 2023 at 4:49 AM Masahiko Sawada <[email protected]> wrote:
> >
> > Hi,
> >
> > I've read this thread and the latest patch. IIUC with SAVE_ERROR
> > option, COPY FROM creates an error table for the target table and
> > writes error information there.
> >
> > While I agree that the final shape of this feature would be something
> > like that design, I'm concerned some features are missing in order to
> > make this feature useful in practice. For instance, error logs are
> > inserted to error tables without bounds, meaning that users who want
> > to tolerate errors during COPY FROM will have to truncate or drop the
> > error tables periodically, or the database will grow with error logs
> > without limit. Ideally such maintenance work should be done by the
> > database. There might be some users who want to log such conversion
> > errors in server logs to avoid such maintenance work. I think we
> > should provide an option for where to write, at least. Also, since the
> > error tables are normal user tables internally, error logs are also
> > replicated to subscribers if there is a publication FOR ALL TABLES,
> > unlike system catalogs. I think some users would not like such
> > behavior.
>
> save the error metadata to system catalogs would be more expensive,
> please see below explanation.
> I have no knowledge of publications.
> but i feel there is a feature request: publication FOR ALL TABLES
> exclude regex_pattern.
> Anyway, that would be another topic.
I don't think the new regex idea would be a good solution for the
existing users who are using FOR ALL TABLES publication. It's not
desirable that they have to change the publication because of this
feature. With the current patch, a logical replication using FOR ALL
TABLES publication will stop immediately after an error information is
inserted into a new error table unless the same error table is created
on subscribers.
>
> > Looking at SAVE_ERROR feature closely, I think it consists of two
> > separate features. That is, it enables COPY FROM to load data while
> > (1) tolerating errors and (2) logging errors to somewhere (i.e., an
> > error table). If we implement only (1), it would be like COPY FROM
> > tolerate errors infinitely and log errors to /dev/null. The user
> > cannot see the error details but I guess it could still help some
> > cases as Andres mentioned[1] (it might be a good idea to send the
> > number of rows successfully loaded in a NOTICE message if some rows
> > could not be loaded). Then with (2), COPY FROM can log error
> > information to somewhere such as tables and server logs and the user
> > can select it. So I'm thinking we may be able to implement this
> > feature incrementally. The first step would be something like an
> > option to ignore all errors or an option to specify the maximum number
> > of errors to tolerate before raising an ERROR. The second step would
> > be to support logging destinations such as server logs and tables.
> >
> > Regards,
> >
> > [1] https://www.postgresql.org/message-id/20231109002600.fuihn34bjqqgmbjm%40awork3.anarazel.de
> >
> > --
> > Masahiko Sawada
> > Amazon Web Services: https://aws.amazon.com
>
> > feature incrementally. The first step would be something like an
> > option to ignore all errors or an option to specify the maximum number
> > of errors to tolerate before raising an ERROR. The second step would
>
> I don't think "specify the maximum number of errors to tolerate
> before raising an ERROR." is very useful....
>
> QUOTE from [1]
> MAXERROR [AS] error_count
> If the load returns the error_count number of errors or greater, the
> load fails. If the load returns fewer errors, it continues and returns
> an INFO message that states the number of rows that could not be
> loaded. Use this parameter to allow loads to continue when certain
> rows fail to load into the table because of formatting errors or other
> inconsistencies in the data.
> Set this value to 0 or 1 if you want the load to fail as soon as the
> first error occurs. The AS keyword is optional. The MAXERROR default
> value is 0 and the limit is 100000.
> The actual number of errors reported might be greater than the
> specified MAXERROR because of the parallel nature of Amazon Redshift.
> If any node in the Amazon Redshift cluster detects that MAXERROR has
> been exceeded, each node reports all of the errors it has encountered.
> END OF QUOTE
>
> option MAXERROR error_count. iiuc, it fails while validating line
> error_count + 1, else it raises a notice, tells you how many rows have
> errors.
>
> * case when error_count is small, and the copy fails, it only tells
> you that at least the error_count line has malformed data. but what if
> the actual malformed rows are very big. In this case, this failure
> error message is not that helpful.
> * case when error_count is very big, and the copy does not fail. then
> the actual malformed data rows are very big (still less than
> error_count). but there is no error report, you don't know which line
> has an error.
>
> Either way, if the file has a large portion of malformed rows, then
> the MAXERROR option does not make sense.
> so maybe we don't need a threshold for tolerating errors.
>
> however, we can have an option, not actually copy to the table, but
> only validate, similar to NOLOAD in [1]
I'm fine even if the feature is not like MAXERROR. If we want a
feature to tolerate errors during COPY FROM, I just thought it might
be a good idea to have a tuning knob for better flexibility, not just
like a on/off switch.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 04:33 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) [email protected]
2023-11-14 10:10 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-11-24 03:52 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-05 10:07 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-06 10:47 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-08 07:09 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-10 10:32 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-11 14:05 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alena Rybakina <[email protected]>
2023-12-12 13:04 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) jian he <[email protected]>
2023-12-14 20:48 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Masahiko Sawada <[email protected]>
@ 2023-12-18 02:41 ` torikoshia <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: torikoshia @ 2023-12-18 02:41 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: jian he <[email protected]>; Alena Rybakina <[email protected]>; Damir Belyalov <[email protected]>; [email protected]; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; [email protected]; Andrei Lepikhov <[email protected]>
On 2023-12-15 05:48, Masahiko Sawada wrote:
Thanks for joining this discussion!
> I've read this thread and the latest patch. IIUC with SAVE_ERROR
> option, COPY FROM creates an error table for the target table and
> writes error information there.
>
> While I agree that the final shape of this feature would be something
> like that design, I'm concerned some features are missing in order to
> make this feature useful in practice. For instance, error logs are
> inserted to error tables without bounds, meaning that users who want
> to tolerate errors during COPY FROM will have to truncate or drop the
> error tables periodically, or the database will grow with error logs
> without limit. Ideally such maintenance work should be done by the
> database. There might be some users who want to log such conversion
> errors in server logs to avoid such maintenance work. I think we
> should provide an option for where to write, at least. Also, since the
> error tables are normal user tables internally, error logs are also
> replicated to subscribers if there is a publication FOR ALL TABLES,
> unlike system catalogs. I think some users would not like such
> behavior.
>
> Looking at SAVE_ERROR feature closely, I think it consists of two
> separate features. That is, it enables COPY FROM to load data while
> (1) tolerating errors and (2) logging errors to somewhere (i.e., an
> error table). If we implement only (1), it would be like COPY FROM
> tolerate errors infinitely and log errors to /dev/null. The user
> cannot see the error details but I guess it could still help some
> cases as Andres mentioned[1] (it might be a good idea to send the
> number of rows successfully loaded in a NOTICE message if some rows
> could not be loaded). Then with (2), COPY FROM can log error
> information to somewhere such as tables and server logs and the user
> can select it.
+1.
I may be biased since I wrote some ~v6 patches which just output the
soft errors and number of skipped rows to log, but I think just (1)
would be worth implementing as you pointed out and I like if users could
choose where to log output.
I think there would be situations where it is preferable to save errors
to server log even considering problems which were pointed out in [1],
i.e. manually loading data.
[1]
https://www.postgresql.org/message-id/739953.1699467519%40sss.pgh.pa.us
> feature incrementally. The first step would be something like an
> option to ignore all errors or an option to specify the maximum number
> of errors to tolerate before raising an ERROR. The second step would
> be to support logging destinations such as server logs and tables.
>
> Regards,
>
> [1]
> https://www.postgresql.org/message-id/20231109002600.fuihn34bjqqgmbjm%40awork3.anarazel.de
--
Regards,
--
Atsushi Torikoshi
NTT DATA Group Corporation
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 19:34 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2023-11-16 00:00 ` jian he <[email protected]>
1 sibling, 0 replies; 117+ messages in thread
From: jian he @ 2023-11-16 00:00 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Andres Freund <[email protected]>; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
On Thu, Nov 9, 2023 at 4:12 AM Tom Lane <[email protected]> wrote:
>
> Daniel Gustafsson <[email protected]> writes:
> >> On 8 Nov 2023, at 19:18, Tom Lane <[email protected]> wrote:
> >> I think an actually usable feature of this sort would involve
> >> copying all the failed lines to some alternate output medium,
> >> perhaps a second table with a TEXT column to receive the original
> >> data line. (Or maybe an array of text that could receive the
> >> broken-down field values?) Maybe we could dump the message info,
> >> line number, field name etc into additional columns.
>
> > I agree that the errors should be easily visible to the user in some way. The
> > feature is for sure interesting, especially in data warehouse type jobs where
> > dirty data is often ingested.
>
> I agree it's interesting, but we need to get it right the first time.
>
> Here is a very straw-man-level sketch of what I think might work.
> The option to COPY FROM looks something like
>
> ERRORS TO other_table_name (item [, item [, ...]])
>
> where the "items" are keywords identifying the information item
> we will insert into each successive column of the target table.
> This design allows the user to decide which items are of use
> to them. I envision items like
>
> LINENO bigint COPY line number, counting from 1
> LINE text raw text of line (after encoding conversion)
> FIELDS text[] separated, de-escaped string fields (the data
> that was or would be fed to input functions)
> FIELD text name of troublesome field, if field-specific
> MESSAGE text error message text
> DETAIL text error message detail, if any
> SQLSTATE text error SQLSTATE code
>
just
SAVE ERRORS
automatically create a table to hold the error. (validate
auto-generated table name uniqueness, validate create privilege).
and the table will have the above related info. if no error then table
gets dropped.
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2023-11-08 23:53 ` Andres Freund <[email protected]>
2023-11-09 00:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
1 sibling, 1 reply; 117+ messages in thread
From: Andres Freund @ 2023-11-08 23:53 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Hi,
On 2023-11-08 13:18:39 -0500, Tom Lane wrote:
> Damir <[email protected]> writes:
> > [ v7-0002-Add-new-COPY-option-IGNORE_DATATYPE_ERRORS.patch ]
>
> Sorry for being so late to the party, but ... I don't think this
> is a well-designed feature as it stands. Simply dropping failed rows
> seems like an unusable definition for any application that has
> pretensions of robustness.
Not everything needs to be a robust application though. I've definitely cursed
at postgres for lacking this.
> I think an actually usable feature of this sort would involve
> copying all the failed lines to some alternate output medium,
> perhaps a second table with a TEXT column to receive the original
> data line. (Or maybe an array of text that could receive the
> broken-down field values?) Maybe we could dump the message info,
> line number, field name etc into additional columns.
If we go in that direction, we should make it possible to *not* use such a
table as well, for some uses it'd be pointless.
Another way of reporting errors could be for copy to return invalid input back
to the client, via the copy protocol. That would allow the client to handle
failing rows and also to abort if the number of errors or the type of errors
gets to be too big.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 23:53 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
@ 2023-11-09 00:00 ` Tom Lane <[email protected]>
2023-11-09 00:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Tom Lane @ 2023-11-09 00:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Andres Freund <[email protected]> writes:
> On 2023-11-08 13:18:39 -0500, Tom Lane wrote:
>> I think an actually usable feature of this sort would involve
>> copying all the failed lines to some alternate output medium,
>> perhaps a second table with a TEXT column to receive the original
>> data line.
> If we go in that direction, we should make it possible to *not* use such a
> table as well, for some uses it'd be pointless.
Why? You can always just drop the errors table if you don't want it.
But I fail to see the use-case for ignoring errors altogether.
> Another way of reporting errors could be for copy to return invalid input back
> to the client, via the copy protocol.
Color me skeptical. There are approximately zero clients in the
world today that could handle simultaneous return of data during
a COPY. Certainly neither libpq nor psql are within hailing
distance of being able to support that. Maybe in some far
future it could be made to work --- but if you want it in the v1
patch, you just moved the goalposts into the next county.
regards, tom lane
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 23:53 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
2023-11-09 00:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
@ 2023-11-09 00:26 ` Andres Freund <[email protected]>
2023-11-09 00:28 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
0 siblings, 1 reply; 117+ messages in thread
From: Andres Freund @ 2023-11-09 00:26 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Damir <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Hi,
On 2023-11-08 19:00:01 -0500, Tom Lane wrote:
> Andres Freund <[email protected]> writes:
> > On 2023-11-08 13:18:39 -0500, Tom Lane wrote:
> >> I think an actually usable feature of this sort would involve
> >> copying all the failed lines to some alternate output medium,
> >> perhaps a second table with a TEXT column to receive the original
> >> data line.
>
> > If we go in that direction, we should make it possible to *not* use such a
> > table as well, for some uses it'd be pointless.
>
> Why? You can always just drop the errors table if you don't want it.
I think it'll often just end up littering the database, particularly if the
callers don't care about a few errors.
> But I fail to see the use-case for ignoring errors altogether.
My experience is that there's often a few errors due to bad encoding, missing
escaping etc that you don't care sufficiently about when importing large
quantities of data.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 117+ messages in thread
* Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) torikoshia <[email protected]>
2023-09-20 16:15 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir <[email protected]>
2023-11-08 18:18 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-08 23:53 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
2023-11-09 00:00 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Tom Lane <[email protected]>
2023-11-09 00:26 ` Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Andres Freund <[email protected]>
@ 2023-11-09 00:28 ` Damir Belyalov <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Damir Belyalov @ 2023-11-09 00:28 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Tom Lane <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; [email protected]; [email protected]; [email protected]; Andrey Lepikhov <[email protected]>; Alena Rybakina <[email protected]>
Hello everyone!
Thanks for turning back to this patch.
I had already thought about storing errors in the table / separate file /
logfile and it seems to me that the best way is to output errors in
logfile. As for user it is more convenient to look for errors in the place
where they are usually generated - in logfile and if he wants to intercept
them he could easily do that by few commands.
The analogues of this feature in other DBSM usually had additional files
for storing errors, but their features had too many options (see attached
files).
I also think that the best way is to simplify this feature for the first
version and don't use redundant adjustments such as additional files and
other options.
IMHO for more complicated operations with loading tables files pgloader
exists: https://github.com/dimitri/pgloader
Links of analogues of COPY IGNORE_DATATYPE_ERRORS
https://dev.mysql.com/doc/refman/8.0/en/load-data.html
https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
Regards,
Damir Belyalov
Postgres Professional
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v7 4/7] Row pattern recognition patch (executor).
@ 2023-09-22 04:53 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 853 ++++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 909 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..84d1b8acaa 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -182,8 +184,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +198,32 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringInfo *str_set, int set_size);
+static void search_str_set_recurse(char *pattern, StringInfo *str_set, int set_size, int set_index,
+ int str_index, char *encoded_str, int *resultlen);
+static char pattern_initial(WindowAggState *winstate, char *vname);
/*
* initialize_windowaggregate
@@ -673,6 +699,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +805,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +818,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +894,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +952,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +988,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1008,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1058,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1079,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1183,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2147,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2317,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2495,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2593,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2787,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2827,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2895,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2946,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3100,7 +3308,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3628,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,11 +3742,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3565,6 +3823,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3847,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3884,572 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+
+ /*
+ * Array of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+ #define ENCODED_STR_ARRAY_ALLOC_SIZE 128
+ StringInfo *str_set = NULL;
+ int str_set_index;
+ int str_set_size;
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ /* build encoded string array */
+ if (str_set == NULL)
+ {
+ str_set_index = 0;
+ str_set_size = ENCODED_STR_ARRAY_ALLOC_SIZE * sizeof(StringInfo);
+ str_set = palloc(str_set_size);
+ }
+
+ str_set[str_set_index++] = encoded_str;
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ if (str_set_index >= str_set_size)
+ {
+ str_set_size *= 2;
+ str_set = repalloc(str_set, str_set_size);
+ }
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (str_set == NULL)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set, str_set_index);
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform regex search pattern against encoded string array str_set.
+ * returns the number of longest matching rows.
+ * str_set: array of encoded string. Each array element corresponds to each
+ * row.
+ * set_size: size of set_str array.
+ */
+static
+int search_str_set(char *pattern, StringInfo *str_set, int set_size)
+{
+ char *encoded_str = palloc0(set_size+1);
+ int resultlen = 0;
+
+ search_str_set_recurse(pattern, str_set, set_size, 0, 0,
+ encoded_str, &resultlen);
+ elog(DEBUG1, "search_str_set returns %d", resultlen);
+ return resultlen;
+}
+
+/*
+ * Workhorse of search_str_set.
+ *
+ * Recurse among matched pattern variables in a row. The max recursion depth
+ * is number of pattern variables matched in a row.
+ */
+static
+void search_str_set_recurse(char *pattern, StringInfo *str_set,
+ int set_size, int set_index, int str_index,
+ char *encoded_str, int *resultlen)
+{
+ for (;;)
+ {
+ char c;
+
+ c = str_set[set_index]->data[str_index];
+ if (c == '\0')
+ return;
+ encoded_str[set_index] = c;
+ set_index++;
+
+ if (set_index >= set_size)
+ {
+ Datum d;
+ text *res;
+ char *substr;
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ int len;
+
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ if (len > *resultlen)
+ /* remember the longest match */
+ *resultlen = len;
+ }
+ }
+ return;
+ }
+ else
+ search_str_set_recurse(pattern, str_set, set_size, set_index, str_index + 1,
+ encoded_str, resultlen);
+ }
+}
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..d20f803cf5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..63feb68f60 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2471,6 +2471,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2519,6 +2524,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2569,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Fri_Sep_22_14_16_40_2023_530)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v7-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v8 4/7] Row pattern recognition patch (executor).
@ 2023-09-25 05:01 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-09-25 05:01 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1024 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1080 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 310ac23e3a..5f2dfdf943 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,12 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +190,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +204,37 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
/*
* initialize_windowaggregate
@@ -673,6 +710,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +816,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +829,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +905,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +963,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +999,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1019,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1069,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1090,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1194,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2158,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2328,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2506,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2604,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2798,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2838,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2691,6 +2906,8 @@ ExecEndWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -2740,6 +2957,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3100,7 +3319,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3420,14 +3639,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3494,11 +3753,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3565,6 +3834,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3583,15 +3858,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3622,3 +3895,732 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+
+ /*
+ * Set of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set);
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. str_set is a set
+ * of StringInfo. Each StringInfo has a string comprising initial characters
+ * of pattern variables being true in a row. Returns the longest number of
+ * the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set)
+{
+ int set_size; /* number of rows in the set */
+ int resultlen = 0;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * loop over each new pattern variable char in previous result
+ * char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ else
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = str->data;
+
+ /*
+ * loop over each pattern variable char in the input set.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* copy source string */
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ /* we no long need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ int len;
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no long need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ }
+ }
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ pfree(str->data);
+ pfree(str);
+ }
+ pfree(string_set);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9805bc6118..d20f803cf5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index cb714f4a19..63feb68f60 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2471,6 +2471,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2519,6 +2524,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2555,6 +2569,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Mon_Sep_25_14_26_30_2023_752)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v8-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v9 4/7] Row pattern recognition patch (executor).
@ 2023-10-04 05:51 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-10-04 05:51 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1021 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1077 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 77724a6daa..5da1bb5a6f 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,12 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +190,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +204,37 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
/*
* initialize_windowaggregate
@@ -673,6 +710,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +816,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +829,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +905,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +963,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +999,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1019,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1069,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1090,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1194,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2158,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2328,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2506,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2604,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2798,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2838,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2938,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3300,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3620,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3734,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3815,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3839,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3876,731 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+
+ /*
+ * Set of pattern variables evaluted to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+ elog(DEBUG1, "vname: %s initial: %c quantifier: %s", vname, initial, quantifier);
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ num_matched_rows = search_str_set(pattern_str->data, str_set);
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. str_set is a set
+ * of StringInfo. Each StringInfo has a string comprising initial characters
+ * of pattern variables being true in a row. Returns the longest number of
+ * the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set)
+{
+ int set_size; /* number of rows in the set */
+ int resultlen = 0;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * loop over each new pattern variable char in previous result
+ * char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ else
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = str->data;
+
+ /*
+ * loop over each pattern variable char in the input set.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* copy source string */
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ p++; /* next pattern variable */
+ }
+ }
+ /* we no long need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ int len;
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no long need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)))) > 0)
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(cstring_to_text(encoded_str)),
+ PointerGetDatum(cstring_to_text(pattern)));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ }
+ }
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ pfree(str->data);
+ pfree(str);
+ }
+ pfree(string_set);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f0b7b9cbd8..24d288336a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10416,6 +10416,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 108d69ba28..28d098b1cf 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Wed_Oct__4_15_03_28_2023_821)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v9-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v10 4/7] Row pattern recognition patch (executor).
@ 2023-10-22 02:22 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-10-22 02:22 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1363 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1419 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 77724a6daa..2e1baef7ea 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,40 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos {
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +218,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +232,43 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+//static int search_str_set(char *pattern, StringSet *str_set, int *initial_orders);
+static int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
+static VariablePos *variable_pos_init(void);
+static void variable_pos_register(VariablePos *variable_pos, char initial, int pos);
+static bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2);
+static int variable_pos_fetch(VariablePos *variable_pos, char initial, int index);
+static void variable_pos_discard(VariablePos *variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +744,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +850,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +863,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +939,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +997,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1033,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1053,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1103,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1124,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1228,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2192,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2362,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2540,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2638,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2832,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2872,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2972,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3334,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3654,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3768,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3849,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3873,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3910,1039 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+ int initial_index;
+ VariablePos *variable_pos;
+
+ /*
+ * Set of pattern variables evaluated to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ int64 result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ elog(DEBUG1, "search_str_set started");
+ num_matched_rows = search_str_set(pattern_str->data, str_set, variable_pos);
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ int64 i;
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. For example, if DEFINE has variables START, UP and
+ * DOWN, PATTERN HAS START, UP and DOWN, then the initials in PATTERN will be
+ * 'a', 'b' and 'c'.
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ elog(DEBUG1, "index: %d", index);
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+ elog(DEBUG1, "discard this old set because shorter match: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "keep this old set: %s", old->data);
+ new = makeStringInfo();
+ appendStringInfoString(new, old->data);
+ string_set_add(new_str_set, new);
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+ elog(DEBUG1, "discard this old set: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "str->data: %s", str->data);
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed
+ * entries that have shorter match length.
+ * Mark them as "discard" so that they are
+ * discarded in the next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size = string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size; new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char = new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /* mark this set to discard in the next round */
+ appendStringInfoChar(new, DISCARD_CHAR);
+ elog(DEBUG1, "add discard char: %s", new->data);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ elog(DEBUG1, "target string: %s", s->data);
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text, *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ pfree(str->data);
+ pfree(str);
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos *variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void variable_pos_register(VariablePos *variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2)
+{
+ int index1, index2;
+ int pos1, pos2;
+
+ for (index1 = 0; ; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0; ; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int variable_pos_fetch(VariablePos *variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * Discard VariablePos
+ */
+static
+void variable_pos_discard(VariablePos *variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index b87a624fb2..9ebcc7b5d2 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
#include "windowapi.h"
@@ -36,11 +39,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -673,7 +684,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -713,3 +724,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index c92d0631a0..ebb017b015 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10425,6 +10425,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 108d69ba28..28d098b1cf 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Sun_Oct_22_11_39_20_2023_140)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v10-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v11 4/7] Row pattern recognition patch (executor).
@ 2023-11-08 06:57 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1420 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1476 insertions(+), 13 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 77724a6daa..ea5e73c969 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,40 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos {
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +218,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +232,42 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
+static VariablePos *variable_pos_init(void);
+static void variable_pos_register(VariablePos *variable_pos, char initial, int pos);
+static bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2);
+static int variable_pos_fetch(VariablePos *variable_pos, char initial, int index);
+static void variable_pos_discard(VariablePos *variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +743,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +849,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +862,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -861,8 +938,10 @@ eval_windowaggregates(WindowAggState *winstate)
* If we created a mark pointer for aggregates, keep it pushed up to frame
* head, so that tuplestore can discard unnecessary rows.
*/
+#ifdef NOT_USED
if (agg_winobj->markptr >= 0)
WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+#endif
/*
* Now restart the aggregates that require it.
@@ -917,6 +996,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1032,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1052,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1102,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1123,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1227,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2191,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2361,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2539,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2637,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2831,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2871,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2971,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3333,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3653,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3767,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3848,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3872,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3909,1097 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos, i;
+
+ /*
+ * Set of pattern variables evaluated to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier.
+ * If it does not, we can just apply the pattern to each row.
+ * If it succeeds, we are done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s", pos, vname);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+ elog(DEBUG1, "expression result is false or out of frame");
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+ elog(DEBUG1, "pattern matched");
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included.
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ elog(DEBUG1, "search_str_set started");
+ num_matched_rows = search_str_set(pattern_str->data, str_set, variable_pos);
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. For example, if DEFINE has variables START, UP and
+ * DOWN, PATTERN HAS START, UP and DOWN, then the initials in PATTERN will be
+ * 'a', 'b' and 'c'.
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ elog(DEBUG1, "index: %d", index);
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+ elog(DEBUG1, "discard this old set because shorter match: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "keep this old set: %s", old->data);
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+ elog(DEBUG1, "discard this old set: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "str->data: %s", str->data);
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed
+ * entries that have shorter match length.
+ * Mark them as "discard" so that they are
+ * discarded in the next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size = string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size; new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char = new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /* mark this set to discard in the next round */
+ appendStringInfoChar(new, DISCARD_CHAR);
+ elog(DEBUG1, "add discard char: %s", new->data);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ elog(DEBUG1, "target string: %s", s->data);
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text, *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos *variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void variable_pos_register(VariablePos *variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2)
+{
+ int index1, index2;
+ int pos1, pos2;
+
+ for (index1 = 0; ; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0; ; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int variable_pos_fetch(VariablePos *variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * Discard VariablePos
+ */
+static
+void variable_pos_discard(VariablePos *variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 0bfbac00d7..a4a11c7f8d 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index f14aed422a..6df54a3cab 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10423,6 +10423,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 5d7f17dee0..d8f92720bc 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Wed_Nov__8_16_37_05_2023_872)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v11-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v12 4/7] Row pattern recognition patch (executor).
@ 2023-12-04 11:23 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2023-12-04 11:23 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1435 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1490 insertions(+), 14 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3258305f57..a093e5dec6 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,40 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos {
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +218,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +232,42 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
+static VariablePos *variable_pos_init(void);
+static void variable_pos_register(VariablePos *variable_pos, char initial, int pos);
+static bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2);
+static int variable_pos_fetch(VariablePos *variable_pos, char initial, int index);
+static void variable_pos_discard(VariablePos *variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +743,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +849,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +862,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -862,7 +939,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1009,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1045,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1065,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1115,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1136,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1240,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2204,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2374,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2552,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2650,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2844,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2884,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2984,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3346,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3666,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3780,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3861,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3885,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3922,1097 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos, i;
+
+ /*
+ * Set of pattern variables evaluated to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier.
+ * If it does not, we can just apply the pattern to each row.
+ * If it succeeds, we are done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s", pos, vname);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+ elog(DEBUG1, "expression result is false or out of frame");
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+ elog(DEBUG1, "pattern matched");
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included.
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ elog(DEBUG1, "search_str_set started");
+ num_matched_rows = search_str_set(pattern_str->data, str_set, variable_pos);
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. For example, if DEFINE has variables START, UP and
+ * DOWN, PATTERN HAS START, UP and DOWN, then the initials in PATTERN will be
+ * 'a', 'b' and 'c'.
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ elog(DEBUG1, "index: %d", index);
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+ elog(DEBUG1, "discard this old set because shorter match: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "keep this old set: %s", old->data);
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+ elog(DEBUG1, "discard this old set: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "str->data: %s", str->data);
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed
+ * entries that have shorter match length.
+ * Mark them as "discard" so that they are
+ * discarded in the next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size = string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size; new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char = new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /* mark this set to discard in the next round */
+ appendStringInfoChar(new, DISCARD_CHAR);
+ elog(DEBUG1, "add discard char: %s", new->data);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ elog(DEBUG1, "target string: %s", s->data);
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text, *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos *variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void variable_pos_register(VariablePos *variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2)
+{
+ int index1, index2;
+ int pos1, pos2;
+
+ for (index1 = 0; ; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0; ; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int variable_pos_fetch(VariablePos *variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * Discard VariablePos
+ */
+static
+void variable_pos_discard(VariablePos *variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 0bfbac00d7..a4a11c7f8d 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fb58dee3bc..aec365cf07 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10437,6 +10437,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 5d7f17dee0..d8f92720bc 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Fri_Dec__8_10_16_13_2023_489)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v12-0005-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v13 5/8] Row pattern recognition patch (executor).
@ 2024-01-22 09:45 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-01-22 09:45 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1435 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 26 +
4 files changed, 1490 insertions(+), 14 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 62d028379b..28d1110b8a 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,40 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet {
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos {
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -182,8 +218,9 @@ static void begin_partition(WindowAggState *winstate);
static void spool_tuples(WindowAggState *winstate, int64 pos);
static void release_partition(WindowAggState *winstate);
-static int row_is_in_frame(WindowAggState *winstate, int64 pos,
+static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +232,42 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
-static bool window_gettupleslot(WindowObject winobj, int64 pos,
- TupleTableSlot *slot);
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
+static bool window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot);
+
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet *string_set_init(void);
+static void string_set_add(StringSet *string_set, StringInfo str);
+static StringInfo string_set_get(StringSet *string_set, int index);
+static int string_set_get_size(StringSet *string_set);
+static void string_set_discard(StringSet *string_set);
+static VariablePos *variable_pos_init(void);
+static void variable_pos_register(VariablePos *variable_pos, char initial, int pos);
+static bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2);
+static int variable_pos_fetch(VariablePos *variable_pos, char initial, int index);
+static void variable_pos_discard(VariablePos *variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +743,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +849,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +862,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -862,7 +939,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1009,29 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate, winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1045,11 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT, winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1065,28 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate, winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate, winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj, winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1115,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1136,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP
+ * TO PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1240,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2204,8 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT , winstate->currentpos);
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2374,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear existing
+ * reduced frame info so that we newly calculate the info starting from
+ * current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2552,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2650,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2844,39 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+ elog(DEBUG1, "defineVariable name: %s", name);
+ winstate->defineVariableList = lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *)expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList = lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2884,57 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *)node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args", func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible that arg type is Const? */
+ var = (Var *)expr;
+
+ if (func->funcid == F_PREV)
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +2984,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3346,7 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT, pos, winobj->markpos );
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3666,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3780,21 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /* RPR cares about frame head pos. Need to call update_frameheadpos */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3861,12 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj, winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos + num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3885,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3922,1097 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame according
+ * to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1; get_reduced_frame_map(winstate,i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT, state, pos);
+ break;
+ }
+
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT, rtn, pos);
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * Create reduced frame map
+ */
+static
+void create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext, REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * Clear reduced frame map
+ */
+static
+void clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * Get reduced frame map specified by pos
+ */
+static
+int get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1, *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int str_set_index = 0;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos, i;
+
+ /*
+ * Set of pattern variables evaluated to true.
+ * Each character corresponds to pattern variable.
+ * Example:
+ * str_set[0] = "AB";
+ * str_set[1] = "AC";
+ * In this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier.
+ * If it does not, we can just apply the pattern to each row.
+ * If it succeeds, we are done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s", pos, vname);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+ elog(DEBUG1, "expression result is false or out of frame");
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+ elog(DEBUG1, "pattern matched");
+
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included.
+ * Loop over until none of pattern matches or encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s", pos, vname, quantifier);
+
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname, encoded_str, &expression_result);
+ if (expression_result)
+ {
+ elog(DEBUG1, "expression result is true");
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " str_set_index: %d encoded_str: %s", pos, str_set_index, encoded_str->data);
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s", pos, encoded_str->data);
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth (lc1, winstate->patternVariableList, lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s", pos, pattern_str->data);
+
+ /* look for matching pattern variable sequence */
+ elog(DEBUG1, "search_str_set started");
+ num_matched_rows = search_str_set(pattern_str->data, str_set, variable_pos);
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * Perform pattern matching using pattern against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. For example, if DEFINE has variables START, UP and
+ * DOWN, PATTERN HAS START, UP and DOWN, then the initials in PATTERN will be
+ * 'a', 'b' and 'c'.
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows.
+ */
+static
+int search_str_set(char *pattern, StringSet *str_set, VariablePos *variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set, *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+ elog(DEBUG1, "index: %d", index);
+
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size ; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+ elog(DEBUG1, "discard this old set because shorter match: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "keep this old set: %s", old->data);
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+ elog(DEBUG1, "discard this old set: %s", old->data);
+ continue;
+ }
+
+ elog(DEBUG1, "str->data: %s", str->data);
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed
+ * entries that have shorter match length.
+ * Mark them as "discard" so that they are
+ * discarded in the next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size = string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size; new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char = new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /* mark this set to discard in the next round */
+ appendStringInfoChar(new, DISCARD_CHAR);
+ elog(DEBUG1, "add discard char: %s", new->data);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+ elog(DEBUG1, "target string: %s", s->data);
+
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text, *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the
+ * matched string is. That is the number of rows in the reduced window
+ * frame. The reason why we can't call textregexsubstr in the first
+ * place is, it errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+
+/*
+ * Evaluate expression associated with PATTERN variable vname.
+ * relpos is relative row position in a frame (starting from 0).
+ * "quantifier" is the quatifier part of the PATTERN regular expression.
+ * Currently only '+' is allowed.
+ * result is out paramater representing the expression evaluation result
+ * is true of false.
+ * Return values are:
+ * >=0: the last match absolute row position
+ * other wise out of frame.
+ */
+static
+int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1, *lc2, *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree (lc1, winstate->defineVariableList, lc2, winstate->defineClauseList, lc3, winstate->defineInitial)
+ {
+ char initial;
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT, vname, current_pos);
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT, vname, current_pos);
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT, current_pos);
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT, current_pos);
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT, current_pos - 1);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT, current_pos - 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT, current_pos + 1);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT, current_pos + 1);
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1, *lc2;
+
+ forboth (lc1, winstate->defineVariableList, lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet *string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void string_set_add(StringSet *string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo string_set_get(StringSet *string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 ||index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * Returns the size of StringSet.
+ */
+static
+int string_set_get_size(StringSet *string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void string_set_discard(StringSet *string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos *variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void variable_pos_register(VariablePos *variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool variable_pos_compare(VariablePos *variable_pos, char initial1, char initial2)
+{
+ int index1, index2;
+ int pos1, pos2;
+
+ for (index1 = 0; ; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0; ; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int variable_pos_fetch(VariablePos *variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * Discard VariablePos
+ */
+static
+void variable_pos_discard(VariablePos *variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 095de7741d..67a890f992 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ad74e07dbb..a34ddaa385 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10452,6 +10452,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 561fdd98f1..888dad4c7d 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2470,6 +2470,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2518,6 +2523,15 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2554,6 +2568,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Mon_Jan_22_19_26_18_2024_011)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v14 5/8] Row pattern recognition patch (executor).
@ 2024-02-28 13:59 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-02-28 13:59 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1617 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1679 insertions(+), 11 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 62d028379b..c473cbd218 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +752,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +858,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +871,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -862,7 +948,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1018,30 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate,
+ winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1055,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT,
+ winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1076,32 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1151,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP TO
+ * PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1255,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2219,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2392,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2570,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2483,6 +2668,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2862,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2906,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3013,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3375,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3696,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3810,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3895,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3921,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3958,1249 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 095de7741d..c5bd28ce19 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/builtins.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9c120fc2b7..81e4580b13 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10456,6 +10456,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 444a5f0fd5..15d8ac4c1e 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2477,6 +2477,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2525,6 +2530,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2561,6 +2579,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Thu_Feb_29_09_19_54_2024_640)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v14-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v15 5/8] Row pattern recognition patch (executor).
@ 2024-03-28 10:30 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-03-28 10:30 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1617 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1679 insertions(+), 11 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..9606e7d463 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +752,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +858,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +871,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -862,7 +948,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1018,30 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate,
+ winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1055,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT,
+ winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1076,32 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1151,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP TO
+ * PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1255,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2219,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2392,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2570,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2671,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2862,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2906,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3013,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3375,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3696,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3810,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3895,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3921,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3958,1249 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07023ee61d..b9d171e327 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10474,6 +10474,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 1774c56ae3..f2322ace6f 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2549,6 +2549,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2597,6 +2602,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2633,6 +2651,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Thu_Mar_28_19_59_25_2024_076)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v16 5/8] Row pattern recognition patch (executor).
@ 2024-04-12 06:49 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-04-12 06:49 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1617 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1679 insertions(+), 11 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..9606e7d463 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -673,6 +752,7 @@ eval_windowaggregates(WindowAggState *winstate)
WindowObject agg_winobj;
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot;
+ bool agg_result_isnull;
numaggs = winstate->numaggs;
if (numaggs == 0)
@@ -778,6 +858,9 @@ eval_windowaggregates(WindowAggState *winstate)
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ *
+ * - if RPR is enabled and skip mode is SKIP TO NEXT ROW,
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,8 +871,11 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ (rpr_is_defined(winstate) &&
+ winstate->rpSkipTo == ST_NEXT_ROW))
{
+ elog(DEBUG1, "peraggstate->restart is set");
peraggstate->restart = true;
numaggs_restart++;
}
@@ -862,7 +948,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1018,30 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
+ }
+
+ agg_result_isnull = false;
+ /* RPR is defined? */
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the skip mode is SKIP TO PAST LAST ROW and we already know that
+ * current row is a skipped row, we don't need to accumulate rows,
+ * just return NULL. Note that for unamtched row, we need to do
+ * aggregation so that count(*) shows 0, rather than NULL.
+ */
+ if (winstate->rpSkipTo == ST_PAST_LAST_ROW &&
+ get_reduced_frame_map(winstate,
+ winstate->currentpos) == RF_SKIPPED)
+ agg_result_isnull = true;
}
/*
@@ -930,6 +1055,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+ elog(DEBUG1, "===== loop in frame starts: " INT64_FORMAT,
+ winstate->aggregatedupto);
+
+ if (agg_result_isnull)
+ break;
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1076,32 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -996,6 +1151,16 @@ next_tuple:
peraggstate,
result, isnull);
+ /*
+ * RPR is defined and we just return NULL because skip mode is SKIP TO
+ * PAST LAST ROW and current row is skipped row.
+ */
+ if (agg_result_isnull)
+ {
+ *isnull = true;
+ *result = (Datum) 0;
+ }
+
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1255,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2219,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2392,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2570,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2671,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2862,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2906,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3013,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3375,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3696,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3810,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3895,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3921,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3958,1249 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 134e3b22fd..5f7fb538f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10477,6 +10477,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index fa2f70b7a4..053ad1764c 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2554,6 +2554,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2602,6 +2607,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2638,6 +2656,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Fri_Apr_12_16_09_08_2024_262)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v16-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v17 5/8] Row pattern recognition patch (executor).
@ 2024-04-28 11:00 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-04-28 11:00 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..140bb3941e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1244,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2208,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2381,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2559,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2660,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2851,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2895,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3002,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3364,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3685,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3799,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3884,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3910,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3947,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 134e3b22fd..5f7fb538f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10477,6 +10477,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index d927ac44a8..971d8682b1 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2550,6 +2550,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2598,6 +2603,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2634,6 +2652,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Sun_Apr_28_20_28_26_2024_444)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v17-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v18 5/8] Row pattern recognition patch (executor).
@ 2024-05-11 07:11 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-05-11 07:11 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..140bb3941e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1244,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2208,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2381,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2559,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2660,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2851,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2895,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3002,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3364,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3685,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3799,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3884,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3910,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3947,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 134e3b22fd..5f7fb538f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10477,6 +10477,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 8bc421e7c0..4dd9a17eca 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2554,6 +2554,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2602,6 +2607,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2638,6 +2656,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Sat_May_11_16_23_07_2024_789)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v18-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v19 5/8] Row pattern recognition patch (executor).
@ 2024-05-14 23:26 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..140bb3941e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1244,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2208,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2381,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2559,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2660,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2851,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2895,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3002,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3364,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3685,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3799,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3884,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3910,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3947,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2a9f2105b1..595ef7502e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10479,6 +10479,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 8bc421e7c0..4dd9a17eca 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2554,6 +2554,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2602,6 +2607,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2638,6 +2656,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Wed_May_15_09_02_03_2024_008)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v19-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v20 5/8] Row pattern recognition patch (executor).
@ 2024-05-24 02:26 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-05-24 02:26 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..140bb3941e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1244,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2208,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2381,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2559,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2660,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2851,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2895,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3002,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3364,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3685,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3799,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3884,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3910,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3947,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6a5476d3c4..5e7506dabb 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10479,6 +10479,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 8bc421e7c0..4dd9a17eca 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2554,6 +2554,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2602,6 +2607,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2638,6 +2656,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Fri_May_24_11_39_19_2024_763)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v20-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v21 5/8] Row pattern recognition patch (executor).
@ 2024-08-26 04:32 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-08-26 04:32 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 3221fa1522..140bb3941e 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1090,6 +1244,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2053,6 +2208,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2221,6 +2381,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2388,6 +2559,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2486,6 +2660,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2667,6 +2851,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2674,6 +2895,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2723,6 +3002,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3083,7 +3364,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3403,14 +3685,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3477,11 +3799,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3548,6 +3884,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3566,15 +3910,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3605,3 +3947,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4abc6d9526..c3fafed291 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10549,6 +10549,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index af7d8fd1e7..609b066845 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2578,6 +2578,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2626,6 +2631,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2662,6 +2680,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Mon_Aug_26_13_39_47_2024_878)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v21-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
* [PATCH v22 5/8] Row pattern recognition patch (executor).
@ 2024-09-19 04:48 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 117+ messages in thread
From: Tatsuo Ishii @ 2024-09-19 04:48 UTC (permalink / raw)
---
src/backend/executor/nodeWindowAgg.c | 1610 +++++++++++++++++++++++++-
src/backend/utils/adt/windowfuncs.c | 37 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/nodes/execnodes.h | 30 +
4 files changed, 1671 insertions(+), 12 deletions(-)
diff --git a/src/backend/executor/nodeWindowAgg.c b/src/backend/executor/nodeWindowAgg.c
index 51a6708a39..e46a3dd1b7 100644
--- a/src/backend/executor/nodeWindowAgg.c
+++ b/src/backend/executor/nodeWindowAgg.c
@@ -36,6 +36,7 @@
#include "access/htup_details.h"
#include "catalog/objectaccess.h"
#include "catalog/pg_aggregate.h"
+#include "catalog/pg_collation_d.h"
#include "catalog/pg_proc.h"
#include "executor/executor.h"
#include "executor/nodeWindowAgg.h"
@@ -48,6 +49,7 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
+#include "utils/fmgroids.h"
#include "utils/expandeddatum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
@@ -159,6 +161,43 @@ typedef struct WindowStatePerAggData
bool restart; /* need to restart this agg in this cycle? */
} WindowStatePerAggData;
+/*
+ * Set of StringInfo. Used in RPR.
+ */
+typedef struct StringSet
+{
+ StringInfo *str_set;
+ Size set_size; /* current array allocation size in number of
+ * items */
+ int set_index; /* current used size */
+} StringSet;
+
+/*
+ * Allowed subsequent PATTERN variables positions.
+ * Used in RPR.
+ *
+ * pos represents the pattern variable defined order in DEFINE caluase. For
+ * example. "DEFINE START..., UP..., DOWN ..." and "PATTERN START UP DOWN UP"
+ * will create:
+ * VariablePos[0].pos[0] = 0; START
+ * VariablePos[1].pos[0] = 1; UP
+ * VariablePos[1].pos[1] = 3; UP
+ * VariablePos[2].pos[0] = 2; DOWN
+ *
+ * Note that UP has two pos because UP appears in PATTERN twice.
+ *
+ * By using this strucrture, we can know which pattern variable can be followed
+ * by which pattern variable(s). For example, START can be followed by UP and
+ * DOWN since START's pos is 0, and UP's pos is 1 or 3, DOWN's pos is 2.
+ * DOWN can be followed by UP since UP's pos is either 1 or 3.
+ *
+ */
+#define NUM_ALPHABETS 26 /* we allow [a-z] variable initials */
+typedef struct VariablePos
+{
+ int pos[NUM_ALPHABETS]; /* postion(s) in PATTERN */
+} VariablePos;
+
static void initialize_windowaggregate(WindowAggState *winstate,
WindowStatePerFunc perfuncstate,
WindowStatePerAgg peraggstate);
@@ -184,6 +223,7 @@ static void release_partition(WindowAggState *winstate);
static int row_is_in_frame(WindowAggState *winstate, int64 pos,
TupleTableSlot *slot);
+
static void update_frameheadpos(WindowAggState *winstate);
static void update_frametailpos(WindowAggState *winstate);
static void update_grouptailpos(WindowAggState *winstate);
@@ -195,9 +235,48 @@ static Datum GetAggInitVal(Datum textInitVal, Oid transtype);
static bool are_peers(WindowAggState *winstate, TupleTableSlot *slot1,
TupleTableSlot *slot2);
+
+static int WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout);
static bool window_gettupleslot(WindowObject winobj, int64 pos,
TupleTableSlot *slot);
+static void attno_map(Node *node);
+static bool attno_map_walker(Node *node, void *context);
+static int row_is_in_reduced_frame(WindowObject winobj, int64 pos);
+static bool rpr_is_defined(WindowAggState *winstate);
+
+static void create_reduced_frame_map(WindowAggState *winstate);
+static int get_reduced_frame_map(WindowAggState *winstate, int64 pos);
+static void register_reduced_frame_map(WindowAggState *winstate, int64 pos,
+ int val);
+static void clear_reduced_frame_map(WindowAggState *winstate);
+static void update_reduced_frame(WindowObject winobj, int64 pos);
+
+static int64 evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result);
+
+static bool get_slots(WindowObject winobj, int64 current_pos);
+
+static int search_str_set(char *pattern, StringSet * str_set,
+ VariablePos * variable_pos);
+static char pattern_initial(WindowAggState *winstate, char *vname);
+static int do_pattern_match(char *pattern, char *encoded_str);
+
+static StringSet * string_set_init(void);
+static void string_set_add(StringSet * string_set, StringInfo str);
+static StringInfo string_set_get(StringSet * string_set, int index);
+static int string_set_get_size(StringSet * string_set);
+static void string_set_discard(StringSet * string_set);
+static VariablePos * variable_pos_init(void);
+static void variable_pos_register(VariablePos * variable_pos, char initial,
+ int pos);
+static bool variable_pos_compare(VariablePos * variable_pos,
+ char initial1, char initial2);
+static int variable_pos_fetch(VariablePos * variable_pos, char initial,
+ int index);
+static void variable_pos_discard(VariablePos * variable_pos);
/*
* initialize_windowaggregate
@@ -774,10 +853,12 @@ eval_windowaggregates(WindowAggState *winstate)
* transition function, or
* - we have an EXCLUSION clause, or
* - if the new frame doesn't overlap the old one
+ * - if RPR is enabled
*
* Note that we don't strictly need to restart in the last case, but if
* we're going to remove all rows from the aggregation anyway, a restart
* surely is faster.
+ * we restart aggregation too.
*----------
*/
numaggs_restart = 0;
@@ -788,7 +869,8 @@ eval_windowaggregates(WindowAggState *winstate)
(winstate->aggregatedbase != winstate->frameheadpos &&
!OidIsValid(peraggstate->invtransfn_oid)) ||
(winstate->frameOptions & FRAMEOPTION_EXCLUSION) ||
- winstate->aggregatedupto <= winstate->frameheadpos)
+ winstate->aggregatedupto <= winstate->frameheadpos ||
+ rpr_is_defined(winstate))
{
peraggstate->restart = true;
numaggs_restart++;
@@ -862,7 +944,22 @@ eval_windowaggregates(WindowAggState *winstate)
* head, so that tuplestore can discard unnecessary rows.
*/
if (agg_winobj->markptr >= 0)
- WinSetMarkPosition(agg_winobj, winstate->frameheadpos);
+ {
+ int64 markpos = winstate->frameheadpos;
+
+ if (rpr_is_defined(winstate))
+ {
+ /*
+ * If RPR is used, it is possible PREV wants to look at the
+ * previous row. So the mark pos should be frameheadpos - 1
+ * unless it is below 0.
+ */
+ markpos -= 1;
+ if (markpos < 0)
+ markpos = 0;
+ }
+ WinSetMarkPosition(agg_winobj, markpos);
+ }
/*
* Now restart the aggregates that require it.
@@ -917,6 +1014,14 @@ eval_windowaggregates(WindowAggState *winstate)
{
winstate->aggregatedupto = winstate->frameheadpos;
ExecClearTuple(agg_row_slot);
+
+ /*
+ * If RPR is defined, we do not use aggregatedupto_nonrestarted. To
+ * avoid assertion failure below, we reset aggregatedupto_nonrestarted
+ * to frameheadpos.
+ */
+ if (rpr_is_defined(winstate))
+ aggregatedupto_nonrestarted = winstate->frameheadpos;
}
/*
@@ -930,6 +1035,12 @@ eval_windowaggregates(WindowAggState *winstate)
{
int ret;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "===== loop in frame starts: aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+
/* Fetch next row if we didn't already */
if (TupIsNull(agg_row_slot))
{
@@ -945,9 +1056,52 @@ eval_windowaggregates(WindowAggState *winstate)
ret = row_is_in_frame(winstate, winstate->aggregatedupto, agg_row_slot);
if (ret < 0)
break;
+
if (ret == 0)
goto next_tuple;
+ if (rpr_is_defined(winstate))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "reduced_frame_map: %d aggregatedupto: " INT64_FORMAT " aggregatedbase: " INT64_FORMAT,
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto),
+ winstate->aggregatedupto,
+ winstate->aggregatedbase);
+#endif
+ /*
+ * If the row status at currentpos is already decided and current
+ * row status is not decided yet, it means we passed the last
+ * reduced frame. Time to break the loop.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->currentpos) != RF_NOT_DETERMINED &&
+ get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_NOT_DETERMINED)
+ break;
+
+ /*
+ * Otherwise we need to calculate the reduced frame.
+ */
+ ret = row_is_in_reduced_frame(winstate->agg_winobj,
+ winstate->aggregatedupto);
+ if (ret == -1) /* unmatched row */
+ break;
+
+ /*
+ * Check if current row needs to be skipped due to no match.
+ */
+ if (get_reduced_frame_map(winstate,
+ winstate->aggregatedupto) == RF_SKIPPED &&
+ winstate->aggregatedupto == winstate->aggregatedbase)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "skip current row for aggregation");
+#endif
+ break;
+ }
+ }
+
/* Set tuple context for evaluation of aggregate arguments */
winstate->tmpcontext->ecxt_outertuple = agg_row_slot;
@@ -976,6 +1130,7 @@ next_tuple:
ExecClearTuple(agg_row_slot);
}
+
/* The frame's end is not supposed to move backwards, ever */
Assert(aggregatedupto_nonrestarted <= winstate->aggregatedupto);
@@ -995,7 +1150,6 @@ next_tuple:
&winstate->perfunc[wfuncno],
peraggstate,
result, isnull);
-
/*
* save the result in case next row shares the same frame.
*
@@ -1199,6 +1353,7 @@ begin_partition(WindowAggState *winstate)
winstate->framehead_valid = false;
winstate->frametail_valid = false;
winstate->grouptail_valid = false;
+ create_reduced_frame_map(winstate);
winstate->spooled_rows = 0;
winstate->currentpos = 0;
winstate->frameheadpos = 0;
@@ -2170,6 +2325,11 @@ ExecWindowAgg(PlanState *pstate)
CHECK_FOR_INTERRUPTS();
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "ExecWindowAgg called. pos: " INT64_FORMAT,
+ winstate->currentpos);
+#endif
+
if (winstate->status == WINDOWAGG_DONE)
return NULL;
@@ -2278,6 +2438,17 @@ ExecWindowAgg(PlanState *pstate)
/* don't evaluate the window functions when we're in pass-through mode */
if (winstate->status == WINDOWAGG_RUN)
{
+ /*
+ * If RPR is defined and skip mode is next row, we need to clear
+ * existing reduced frame info so that we newly calculate the info
+ * starting from current row.
+ */
+ if (rpr_is_defined(winstate))
+ {
+ if (winstate->rpSkipTo == ST_NEXT_ROW)
+ clear_reduced_frame_map(winstate);
+ }
+
/*
* Evaluate true window functions
*/
@@ -2445,6 +2616,9 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
TupleDesc scanDesc;
ListCell *l;
+ TargetEntry *te;
+ Expr *expr;
+
/* check for unsupported flags */
Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK)));
@@ -2543,6 +2717,16 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->temp_slot_2 = ExecInitExtraTupleSlot(estate, scanDesc,
&TTSOpsMinimalTuple);
+ winstate->prev_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->next_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+
+ winstate->null_slot = ExecInitExtraTupleSlot(estate, scanDesc,
+ &TTSOpsMinimalTuple);
+ winstate->null_slot = ExecStoreAllNullTuple(winstate->null_slot);
+
/*
* create frame head and tail slots only if needed (must create slots in
* exactly the same cases that update_frameheadpos and update_frametailpos
@@ -2724,6 +2908,43 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
winstate->inRangeAsc = node->inRangeAsc;
winstate->inRangeNullsFirst = node->inRangeNullsFirst;
+ /* Set up SKIP TO type */
+ winstate->rpSkipTo = node->rpSkipTo;
+ /* Set up row pattern recognition PATTERN clause */
+ winstate->patternVariableList = node->patternVariable;
+ winstate->patternRegexpList = node->patternRegexp;
+
+ /* Set up row pattern recognition DEFINE clause */
+ winstate->defineInitial = node->defineInitial;
+ winstate->defineVariableList = NIL;
+ winstate->defineClauseList = NIL;
+ if (node->defineClause != NIL)
+ {
+ /*
+ * Tweak arg var of PREV/NEXT so that it refers to scan/inner slot.
+ */
+ foreach(l, node->defineClause)
+ {
+ char *name;
+ ExprState *exps;
+
+ te = lfirst(l);
+ name = te->resname;
+ expr = te->expr;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "defineVariable name: %s", name);
+#endif
+ winstate->defineVariableList =
+ lappend(winstate->defineVariableList,
+ makeString(pstrdup(name)));
+ attno_map((Node *) expr);
+ exps = ExecInitExpr(expr, (PlanState *) winstate);
+ winstate->defineClauseList =
+ lappend(winstate->defineClauseList, exps);
+ }
+ }
+
winstate->all_first = true;
winstate->partition_spooled = false;
winstate->more_partitions = false;
@@ -2732,6 +2953,64 @@ ExecInitWindowAgg(WindowAgg *node, EState *estate, int eflags)
return winstate;
}
+/*
+ * Rewrite varno of Var node that is the argument of PREV/NET so that it sees
+ * scan tuple (PREV) or inner tuple (NEXT).
+ */
+static void
+attno_map(Node *node)
+{
+ (void) expression_tree_walker(node, attno_map_walker, NULL);
+}
+
+static bool
+attno_map_walker(Node *node, void *context)
+{
+ FuncExpr *func;
+ int nargs;
+ Expr *expr;
+ Var *var;
+
+ if (node == NULL)
+ return false;
+
+ if (IsA(node, FuncExpr))
+ {
+ func = (FuncExpr *) node;
+
+ if (func->funcid == F_PREV || func->funcid == F_NEXT)
+ {
+ /* sanity check */
+ nargs = list_length(func->args);
+ if (list_length(func->args) != 1)
+ elog(ERROR, "PREV/NEXT must have 1 argument but function %d has %d args",
+ func->funcid, nargs);
+
+ expr = (Expr *) lfirst(list_head(func->args));
+ if (!IsA(expr, Var))
+ elog(ERROR, "PREV/NEXT's arg is not Var"); /* XXX: is it possible
+ * that arg type is
+ * Const? */
+ var = (Var *) expr;
+
+ if (func->funcid == F_PREV)
+
+ /*
+ * Rewrite varno from OUTER_VAR to regular var no so that the
+ * var references scan tuple.
+ */
+ var->varno = var->varnosyn;
+ else
+ var->varno = INNER_VAR;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "PREV/NEXT's varno is rewritten to: %d", var->varno);
+#endif
+ }
+ }
+ return expression_tree_walker(node, attno_map_walker, NULL);
+}
+
/* -----------------
* ExecEndWindowAgg
* -----------------
@@ -2789,6 +3068,8 @@ ExecReScanWindowAgg(WindowAggState *node)
ExecClearTuple(node->agg_row_slot);
ExecClearTuple(node->temp_slot_1);
ExecClearTuple(node->temp_slot_2);
+ ExecClearTuple(node->prev_slot);
+ ExecClearTuple(node->next_slot);
if (node->framehead_slot)
ExecClearTuple(node->framehead_slot);
if (node->frametail_slot)
@@ -3149,7 +3430,8 @@ window_gettupleslot(WindowObject winobj, int64 pos, TupleTableSlot *slot)
return false;
if (pos < winobj->markpos)
- elog(ERROR, "cannot fetch row before WindowObject's mark position");
+ elog(ERROR, "cannot fetch row: " INT64_FORMAT " before WindowObject's mark position: " INT64_FORMAT,
+ pos, winobj->markpos);
oldcontext = MemoryContextSwitchTo(winstate->ss.ps.ps_ExprContext->ecxt_per_query_memory);
@@ -3469,14 +3751,54 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
WindowAggState *winstate;
ExprContext *econtext;
TupleTableSlot *slot;
- int64 abs_pos;
- int64 mark_pos;
Assert(WindowObjectIsValid(winobj));
winstate = winobj->winstate;
econtext = winstate->ss.ps.ps_ExprContext;
slot = winstate->temp_slot_1;
+ if (WinGetSlotInFrame(winobj, slot,
+ relpos, seektype, set_mark,
+ isnull, isout) == 0)
+ {
+ econtext->ecxt_outertuple = slot;
+ return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
+ econtext, isnull);
+ }
+
+ if (isout)
+ *isout = true;
+ *isnull = true;
+ return (Datum) 0;
+}
+
+/*
+ * WinGetSlotInFrame
+ * slot: TupleTableSlot to store the result
+ * relpos: signed rowcount offset from the seek position
+ * seektype: WINDOW_SEEK_HEAD or WINDOW_SEEK_TAIL
+ * set_mark: If the row is found/in frame and set_mark is true, the mark is
+ * moved to the row as a side-effect.
+ * isnull: output argument, receives isnull status of result
+ * isout: output argument, set to indicate whether target row position
+ * is out of frame (can pass NULL if caller doesn't care about this)
+ *
+ * Returns 0 if we successfullt got the slot. false if out of frame.
+ * (also isout is set)
+ */
+static int
+WinGetSlotInFrame(WindowObject winobj, TupleTableSlot *slot,
+ int relpos, int seektype, bool set_mark,
+ bool *isnull, bool *isout)
+{
+ WindowAggState *winstate;
+ int64 abs_pos;
+ int64 mark_pos;
+ int num_reduced_frame;
+
+ Assert(WindowObjectIsValid(winobj));
+ winstate = winobj->winstate;
+
switch (seektype)
{
case WINDOW_SEEK_CURRENT:
@@ -3543,11 +3865,25 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
winstate->frameOptions);
break;
}
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ if (relpos >= num_reduced_frame)
+ goto out_of_frame;
break;
case WINDOW_SEEK_TAIL:
/* rejecting relpos > 0 is easy and simplifies code below */
if (relpos > 0)
goto out_of_frame;
+
+ /*
+ * RPR cares about frame head pos. Need to call
+ * update_frameheadpos
+ */
+ update_frameheadpos(winstate);
+
update_frametailpos(winstate);
abs_pos = winstate->frametailpos - 1 + relpos;
@@ -3614,6 +3950,14 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
mark_pos = 0; /* keep compiler quiet */
break;
}
+
+ num_reduced_frame = row_is_in_reduced_frame(winobj,
+ winstate->frameheadpos + relpos);
+ if (num_reduced_frame < 0)
+ goto out_of_frame;
+ else if (num_reduced_frame > 0)
+ abs_pos = winstate->frameheadpos + relpos +
+ num_reduced_frame - 1;
break;
default:
elog(ERROR, "unrecognized window seek type: %d", seektype);
@@ -3632,15 +3976,13 @@ WinGetFuncArgInFrame(WindowObject winobj, int argno,
*isout = false;
if (set_mark)
WinSetMarkPosition(winobj, mark_pos);
- econtext->ecxt_outertuple = slot;
- return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
- econtext, isnull);
+ return 0;
out_of_frame:
if (isout)
*isout = true;
*isnull = true;
- return (Datum) 0;
+ return -1;
}
/*
@@ -3671,3 +4013,1251 @@ WinGetFuncArgCurrent(WindowObject winobj, int argno, bool *isnull)
return ExecEvalExpr((ExprState *) list_nth(winobj->argstates, argno),
econtext, isnull);
}
+
+/*
+ * rpr_is_defined
+ * return true if Row pattern recognition is defined.
+ */
+static
+bool
+rpr_is_defined(WindowAggState *winstate)
+{
+ return winstate->patternVariableList != NIL;
+}
+
+/*
+ * -----------------
+ * row_is_in_reduced_frame
+ * Determine whether a row is in the current row's reduced window frame
+ * according to row pattern matching
+ *
+ * The row must has been already determined that it is in a full window frame
+ * and fetched it into slot.
+ *
+ * Returns:
+ * = 0, RPR is not defined.
+ * >0, if the row is the first in the reduced frame. Return the number of rows
+ * in the reduced frame.
+ * -1, if the row is unmatched row
+ * -2, if the row is in the reduced frame but needed to be skipped because of
+ * AFTER MATCH SKIP PAST LAST ROW
+ * -----------------
+ */
+static
+int
+row_is_in_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ int state;
+ int rtn;
+
+ if (!rpr_is_defined(winstate))
+ {
+ /*
+ * RPR is not defined. Assume that we are always in the the reduced
+ * window frame.
+ */
+ rtn = 0;
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ if (state == RF_NOT_DETERMINED)
+ {
+ update_frameheadpos(winstate);
+ update_reduced_frame(winobj, pos);
+ }
+
+ state = get_reduced_frame_map(winstate, pos);
+
+ switch (state)
+ {
+ int64 i;
+ int num_reduced_rows;
+
+ case RF_FRAME_HEAD:
+ num_reduced_rows = 1;
+ for (i = pos + 1;
+ get_reduced_frame_map(winstate, i) == RF_SKIPPED; i++)
+ num_reduced_rows++;
+ rtn = num_reduced_rows;
+ break;
+
+ case RF_SKIPPED:
+ rtn = -2;
+ break;
+
+ case RF_UNMATCHED:
+ rtn = -1;
+ break;
+
+ default:
+ elog(ERROR, "Unrecognized state: %d at: " INT64_FORMAT,
+ state, pos);
+ break;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "row_is_in_reduced_frame returns %d: pos: " INT64_FORMAT,
+ rtn, pos);
+#endif
+ return rtn;
+}
+
+#define REDUCED_FRAME_MAP_INIT_SIZE 1024L
+
+/*
+ * create_reduced_frame_map
+ * Create reduced frame map
+ */
+static
+void
+create_reduced_frame_map(WindowAggState *winstate)
+{
+ winstate->reduced_frame_map =
+ MemoryContextAlloc(winstate->partcontext,
+ REDUCED_FRAME_MAP_INIT_SIZE);
+ winstate->alloc_sz = REDUCED_FRAME_MAP_INIT_SIZE;
+ clear_reduced_frame_map(winstate);
+}
+
+/*
+ * clear_reduced_frame_map
+ * Clear reduced frame map
+ */
+static
+void
+clear_reduced_frame_map(WindowAggState *winstate)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+ MemSet(winstate->reduced_frame_map, RF_NOT_DETERMINED,
+ winstate->alloc_sz);
+}
+
+/*
+ * get_reduced_frame_map
+ * Get reduced frame map specified by pos
+ */
+static
+int
+get_reduced_frame_map(WindowAggState *winstate, int64 pos)
+{
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0 || pos >= winstate->alloc_sz)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ return winstate->reduced_frame_map[pos];
+}
+
+/*
+ * register_reduced_frame_map
+ * Add/replace reduced frame map member at pos.
+ * If there's no enough space, expand the map.
+ */
+static
+void
+register_reduced_frame_map(WindowAggState *winstate, int64 pos, int val)
+{
+ int64 realloc_sz;
+
+ Assert(winstate->reduced_frame_map != NULL);
+
+ if (pos < 0)
+ elog(ERROR, "wrong pos: " INT64_FORMAT, pos);
+
+ if (pos > winstate->alloc_sz - 1)
+ {
+ realloc_sz = winstate->alloc_sz * 2;
+
+ winstate->reduced_frame_map =
+ repalloc(winstate->reduced_frame_map, realloc_sz);
+
+ MemSet(winstate->reduced_frame_map + winstate->alloc_sz,
+ RF_NOT_DETERMINED, realloc_sz - winstate->alloc_sz);
+
+ winstate->alloc_sz = realloc_sz;
+ }
+
+ winstate->reduced_frame_map[pos] = val;
+}
+
+/*
+ * update_reduced_frame
+ * Update reduced frame info.
+ */
+static
+void
+update_reduced_frame(WindowObject winobj, int64 pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ListCell *lc1,
+ *lc2;
+ bool expression_result;
+ int num_matched_rows;
+ int64 original_pos;
+ bool anymatch;
+ StringInfo encoded_str;
+ StringInfo pattern_str = makeStringInfo();
+ StringSet *str_set;
+ int initial_index;
+ VariablePos *variable_pos;
+ bool greedy = false;
+ int64 result_pos,
+ i;
+
+ /*
+ * Set of pattern variables evaluated to true. Each character corresponds
+ * to pattern variable. Example: str_set[0] = "AB"; str_set[1] = "AC"; In
+ * this case at row 0 A and B are true, and A and C are true in row 1.
+ */
+
+ /* initialize pattern variables set */
+ str_set = string_set_init();
+
+ /* save original pos */
+ original_pos = pos;
+
+ /*
+ * Check if the pattern does not include any greedy quantifier. If it does
+ * not, we can just apply the pattern to each row. If it succeeds, we are
+ * done.
+ */
+ foreach(lc1, winstate->patternRegexpList)
+ {
+ char *quantifier = strVal(lfirst(lc1));
+
+ if (*quantifier == '+' || *quantifier == '*')
+ {
+ greedy = true;
+ break;
+ }
+ }
+
+ /*
+ * Non greedy case
+ */
+ if (!greedy)
+ {
+ num_matched_rows = 0;
+
+ foreach(lc1, winstate->patternVariableList)
+ {
+ char *vname = strVal(lfirst(lc1));
+
+ encoded_str = makeStringInfo();
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s",
+ pos, vname);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (!expression_result || result_pos < 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is false or out of frame");
+#endif
+ register_reduced_frame_map(winstate, original_pos,
+ RF_UNMATCHED);
+ return;
+ }
+ /* move to next row */
+ pos++;
+ num_matched_rows++;
+ }
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern matched");
+#endif
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ return;
+ }
+
+ /*
+ * Greedy quantifiers included. Loop over until none of pattern matches or
+ * encounters end of frame.
+ */
+ for (;;)
+ {
+ result_pos = -1;
+
+ /*
+ * Loop over each PATTERN variable.
+ */
+ anymatch = false;
+ encoded_str = makeStringInfo();
+
+ forboth(lc1, winstate->patternVariableList, lc2,
+ winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+#ifdef RPR_DEBUG
+ char *quantifier = strVal(lfirst(lc2));
+
+ elog(DEBUG1, "pos: " INT64_FORMAT " pattern vname: %s quantifier: %s",
+ pos, vname, quantifier);
+#endif
+ expression_result = false;
+
+ /* evaluate row pattern against current row */
+ result_pos = evaluate_pattern(winobj, pos, vname,
+ encoded_str, &expression_result);
+ if (expression_result)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression result is true");
+#endif
+ anymatch = true;
+ }
+
+ /*
+ * If out of frame, we are done.
+ */
+ if (result_pos < 0)
+ break;
+ }
+
+ if (!anymatch)
+ {
+ /* none of patterns matched. */
+ break;
+ }
+
+ string_set_add(str_set, encoded_str);
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pos: " INT64_FORMAT " encoded_str: %s",
+ encoded_str->data);
+#endif
+
+ /* move to next row */
+ pos++;
+
+ if (result_pos < 0)
+ {
+ /* out of frame */
+ break;
+ }
+ }
+
+ if (string_set_get_size(str_set) == 0)
+ {
+ /* no match found in the first row */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ return;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " encoded_str: %s",
+ pos, encoded_str->data);
+#endif
+
+ /* build regular expression */
+ pattern_str = makeStringInfo();
+ appendStringInfoChar(pattern_str, '^');
+ initial_index = 0;
+
+ variable_pos = variable_pos_init();
+
+ forboth(lc1, winstate->patternVariableList,
+ lc2, winstate->patternRegexpList)
+ {
+ char *vname = strVal(lfirst(lc1));
+ char *quantifier = strVal(lfirst(lc2));
+ char initial;
+
+ initial = pattern_initial(winstate, vname);
+ Assert(initial != 0);
+ appendStringInfoChar(pattern_str, initial);
+ if (quantifier[0])
+ appendStringInfoChar(pattern_str, quantifier[0]);
+
+ /*
+ * Register the initial at initial_index. If the initial appears more
+ * than once, all of it's initial_index will be recorded. This could
+ * happen if a pattern variable appears in the PATTERN clause more
+ * than once like "UP DOWN UP" "UP UP UP".
+ */
+ variable_pos_register(variable_pos, initial, initial_index);
+
+ initial_index++;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG2, "pos: " INT64_FORMAT " pattern: %s",
+ pos, pattern_str->data);
+#endif
+
+ /* look for matching pattern variable sequence */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set started");
+#endif
+ num_matched_rows = search_str_set(pattern_str->data,
+ str_set, variable_pos);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "search_str_set returns: %d", num_matched_rows);
+#endif
+ variable_pos_discard(variable_pos);
+ string_set_discard(str_set);
+
+ /*
+ * We are at the first row in the reduced frame. Save the number of
+ * matched rows as the number of rows in the reduced frame.
+ */
+ if (num_matched_rows <= 0)
+ {
+ /* no match */
+ register_reduced_frame_map(winstate, original_pos, RF_UNMATCHED);
+ }
+ else
+ {
+ register_reduced_frame_map(winstate, original_pos, RF_FRAME_HEAD);
+
+ for (i = original_pos + 1; i < original_pos + num_matched_rows; i++)
+ {
+ register_reduced_frame_map(winstate, i, RF_SKIPPED);
+ }
+ }
+
+ return;
+}
+
+/*
+ * search_str_set
+ * Perform pattern matching using "pattern" against str_set. pattern is a
+ * regular expression derived from PATTERN clause. Note that the regular
+ * expression string is prefixed by '^' and followed by initials represented
+ * in a same way as str_set. str_set is a set of StringInfo. Each StringInfo
+ * has a string comprising initials of pattern variable strings being true in
+ * a row. The initials are one of [a-y], parallel to the order of variable
+ * names in DEFINE clause. Suppose DEFINE has variables START, UP and DOWN. If
+ * PATTERN has START, UP+ and DOWN, then the initials in PATTERN will be 'a',
+ * 'b' and 'c'. The "pattern" will be "^ab+c".
+ *
+ * variable_pos is an array representing the order of pattern variable string
+ * initials in PATTERN clause. For example initial 'a' potion is in
+ * variable_pos[0].pos[0] = 0. Note that if the pattern is "START UP DOWN UP"
+ * (UP appears twice), then "UP" (initial is 'b') has two position 1 and
+ * 3. Thus variable_pos for b is variable_pos[1].pos[0] = 1 and
+ * variable_pos[1].pos[1] = 3.
+ *
+ * Returns the longest number of the matching rows (greedy matching) if
+ * quatifier '+' or '*' is included in "pattern".
+ */
+static
+int
+search_str_set(char *pattern, StringSet * str_set, VariablePos * variable_pos)
+{
+#define MAX_CANDIDATE_NUM 10000 /* max pattern match candidate size */
+#define FREEZED_CHAR 'Z' /* a pattern is freezed if it ends with the
+ * char */
+#define DISCARD_CHAR 'z' /* a pattern is not need to keep */
+
+ int set_size; /* number of rows in the set */
+ int resultlen;
+ int index;
+ StringSet *old_str_set,
+ *new_str_set;
+ int new_str_size;
+ int len;
+
+ set_size = string_set_get_size(str_set);
+ new_str_set = string_set_init();
+ len = 0;
+ resultlen = 0;
+
+ /*
+ * Generate all possible pattern variable name initials as a set of
+ * StringInfo named "new_str_set". For example, if we have two rows
+ * having "ab" (row 0) and "ac" (row 1) in the input str_set, new_str_set
+ * will have set of StringInfo "aa", "ac", "ba" and "bc" in the end.
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern: %s set_size: %d", pattern, set_size);
+#endif
+ for (index = 0; index < set_size; index++)
+ {
+ StringInfo str; /* search target row */
+ char *p;
+ int old_set_size;
+ int i;
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "index: %d", index);
+#endif
+ if (index == 0)
+ {
+ /* copy variables in row 0 */
+ str = string_set_get(str_set, index);
+ p = str->data;
+
+ /*
+ * Loop over each new pattern variable char.
+ */
+ while (*p)
+ {
+ StringInfo new = makeStringInfo();
+
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: NULL new_str: %s", new->data);
+#endif
+ p++; /* next pattern variable */
+ }
+ }
+ else /* index != 0 */
+ {
+ old_str_set = new_str_set;
+ new_str_set = string_set_init();
+ str = string_set_get(str_set, index);
+ old_set_size = string_set_get_size(old_str_set);
+
+ /*
+ * Loop over each rows in the previous result set.
+ */
+ for (i = 0; i < old_set_size; i++)
+ {
+ StringInfo new;
+ char last_old_char;
+ int old_str_len;
+ StringInfo old = string_set_get(old_str_set, i);
+
+ p = old->data;
+ old_str_len = strlen(p);
+ if (old_str_len > 0)
+ last_old_char = p[old_str_len - 1];
+ else
+ last_old_char = '\0';
+
+ /* Is this old set freezed? */
+ if (last_old_char == FREEZED_CHAR)
+ {
+ /* if shorter match. we can discard it */
+ if ((old_str_len - 1) < resultlen)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set because shorter match: %s",
+ old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "keep this old set: %s", old->data);
+#endif
+
+ /* move the old set to new_str_set */
+ string_set_add(new_str_set, old);
+ old_str_set->str_set[i] = NULL;
+ continue;
+ }
+ /* Can this old set be discarded? */
+ else if (last_old_char == DISCARD_CHAR)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this old set: %s", old->data);
+#endif
+ continue;
+ }
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "str->data: %s", str->data);
+#endif
+
+ /*
+ * loop over each pattern variable initial char in the input
+ * set.
+ */
+ for (p = str->data; *p; p++)
+ {
+ /*
+ * Optimization. Check if the row's pattern variable
+ * initial character position is greater than or equal to
+ * the old set's last pattern variable initial character
+ * position. For example, if the old set's last pattern
+ * variable initials are "ab", then the new pattern
+ * variable initial can be "b" or "c" but can not be "a",
+ * if the initials in PATTERN is something like "a b c" or
+ * "a b+ c+" etc. This optimization is possible when we
+ * only allow "+" quantifier.
+ */
+ if (variable_pos_compare(variable_pos, last_old_char, *p))
+ {
+ /* copy source string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add pattern variable char */
+ appendStringInfoChar(new, *p);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s",
+ old->data, new->data);
+#endif
+
+ /*
+ * Adhoc optimization. If the first letter in the
+ * input string is the first and second position one
+ * and there's no associated quatifier '+', then we
+ * can dicard the input because there's no chace to
+ * expand the string further.
+ *
+ * For example, pattern "abc" cannot match "aa".
+ */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "pattern[1]:%c pattern[2]:%c new[0]:%c new[1]:%c",
+ pattern[1], pattern[2], new->data[0], new->data[1]);
+#endif
+ if (pattern[1] == new->data[0] &&
+ pattern[1] == new->data[1] &&
+ pattern[2] != '+' &&
+ pattern[1] != pattern[2])
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "discard this new data: %s",
+ new->data);
+#endif
+ pfree(new->data);
+ pfree(new);
+ continue;
+ }
+
+ /* add new one to string set */
+ string_set_add(new_str_set, new);
+ }
+ else
+ {
+ /*
+ * We are freezing this pattern string. Since there's
+ * no chance to expand the string further, we perform
+ * pattern matching against the string. If it does not
+ * match, we can discard it.
+ */
+ len = do_pattern_match(pattern, old->data);
+
+ if (len <= 0)
+ {
+ /* no match. we can discard it */
+ continue;
+ }
+
+ else if (len <= resultlen)
+ {
+ /* shorter match. we can discard it */
+ continue;
+ }
+ else
+ {
+ /* match length is the longest so far */
+
+ int new_index;
+
+ /* remember the longest match */
+ resultlen = len;
+
+ /* freeze the pattern string */
+ new = makeStringInfo();
+ enlargeStringInfo(new, old->len + 1);
+ appendStringInfoString(new, old->data);
+ /* add freezed mark */
+ appendStringInfoChar(new, FREEZED_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "old_str: %s new_str: %s", old->data, new->data);
+#endif
+ string_set_add(new_str_set, new);
+
+ /*
+ * Search new_str_set to find out freezed entries
+ * that have shorter match length. Mark them as
+ * "discard" so that they are discarded in the
+ * next round.
+ */
+
+ /* new_index_size should be the one before */
+ new_str_size =
+ string_set_get_size(new_str_set) - 1;
+
+ /* loop over new_str_set */
+ for (new_index = 0; new_index < new_str_size;
+ new_index++)
+ {
+ char new_last_char;
+ int new_str_len;
+
+ new = string_set_get(new_str_set, new_index);
+ new_str_len = strlen(new->data);
+ if (new_str_len > 0)
+ {
+ new_last_char =
+ new->data[new_str_len - 1];
+ if (new_last_char == FREEZED_CHAR &&
+ (new_str_len - 1) <= len)
+ {
+ /*
+ * mark this set to discard in the
+ * next round
+ */
+ appendStringInfoChar(new, DISCARD_CHAR);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "add discard char: %s", new->data);
+#endif
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ /* we no longer need old string set */
+ string_set_discard(old_str_set);
+ }
+ }
+
+ /*
+ * Perform pattern matching to find out the longest match.
+ */
+ new_str_size = string_set_get_size(new_str_set);
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "new_str_size: %d", new_str_size);
+#endif
+ len = 0;
+ resultlen = 0;
+
+ for (index = 0; index < new_str_size; index++)
+ {
+ StringInfo s;
+
+ s = string_set_get(new_str_set, index);
+ if (s == NULL)
+ continue; /* no data */
+
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "target string: %s", s->data);
+#endif
+ len = do_pattern_match(pattern, s->data);
+ if (len > resultlen)
+ {
+ /* remember the longest match */
+ resultlen = len;
+
+ /*
+ * If the size of result set is equal to the number of rows in the
+ * set, we are done because it's not possible that the number of
+ * matching rows exceeds the number of rows in the set.
+ */
+ if (resultlen >= set_size)
+ break;
+ }
+ }
+
+ /* we no longer need new string set */
+ string_set_discard(new_str_set);
+
+ return resultlen;
+}
+
+/*
+ * do_pattern_match
+ * perform pattern match using pattern against encoded_str.
+ * returns matching number of rows if matching is succeeded.
+ * Otherwise returns 0.
+ */
+static
+int
+do_pattern_match(char *pattern, char *encoded_str)
+{
+ Datum d;
+ text *res;
+ char *substr;
+ int len = 0;
+ text *pattern_text,
+ *encoded_str_text;
+
+ pattern_text = cstring_to_text(pattern);
+ encoded_str_text = cstring_to_text(encoded_str);
+
+ /*
+ * We first perform pattern matching using regexp_instr, then call
+ * textregexsubstr to get matched substring to know how long the matched
+ * string is. That is the number of rows in the reduced window frame. The
+ * reason why we can't call textregexsubstr in the first place is, it
+ * errors out if pattern does not match.
+ */
+ if (DatumGetInt32(DirectFunctionCall2Coll(
+ regexp_instr, DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text))))
+ {
+ d = DirectFunctionCall2Coll(textregexsubstr,
+ DEFAULT_COLLATION_OID,
+ PointerGetDatum(encoded_str_text),
+ PointerGetDatum(pattern_text));
+ if (d != 0)
+ {
+ res = DatumGetTextPP(d);
+ substr = text_to_cstring(res);
+ len = strlen(substr);
+ pfree(substr);
+ }
+ }
+ pfree(encoded_str_text);
+ pfree(pattern_text);
+
+ return len;
+}
+
+/*
+ * evaluate_pattern
+ * Evaluate expression associated with PATTERN variable vname. current_pos is
+ * relative row position in a frame (starting from 0). If vname is evaluated
+ * to true, initial letters associated with vname is appended to
+ * encode_str. result is out paramater representing the expression evaluation
+ * result is true of false.
+ *---------
+ * Return values are:
+ * >=0: the last match absolute row position
+ * otherwise out of frame.
+ *---------
+ */
+static
+int64
+evaluate_pattern(WindowObject winobj, int64 current_pos,
+ char *vname, StringInfo encoded_str, bool *result)
+{
+ WindowAggState *winstate = winobj->winstate;
+ ExprContext *econtext = winstate->ss.ps.ps_ExprContext;
+ ListCell *lc1,
+ *lc2,
+ *lc3;
+ ExprState *pat;
+ Datum eval_result;
+ bool out_of_frame = false;
+ bool isnull;
+ TupleTableSlot *slot;
+
+ forthree(lc1, winstate->defineVariableList,
+ lc2, winstate->defineClauseList,
+ lc3, winstate->defineInitial)
+ {
+ char initial; /* initial letter associated with vname */
+ char *name = strVal(lfirst(lc1));
+
+ if (strcmp(vname, name))
+ continue;
+
+ initial = *(strVal(lfirst(lc3)));
+
+ /* set expression to evaluate */
+ pat = lfirst(lc2);
+
+ /* get current, previous and next tuples */
+ if (!get_slots(winobj, current_pos))
+ {
+ out_of_frame = true;
+ }
+ else
+ {
+ /* evaluate the expression */
+ eval_result = ExecEvalExpr(pat, econtext, &isnull);
+ if (isnull)
+ {
+ /* expression is NULL */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is NULL at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ if (!DatumGetBool(eval_result))
+ {
+ /* expression is false */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is false at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ *result = false;
+ }
+ else
+ {
+ /* expression is true */
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "expression for %s is true at row: " INT64_FORMAT,
+ vname, current_pos);
+#endif
+ appendStringInfoChar(encoded_str, initial);
+ *result = true;
+ }
+ }
+
+ slot = winstate->temp_slot_1;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->prev_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+ slot = winstate->next_slot;
+ if (slot != winstate->null_slot)
+ ExecClearTuple(slot);
+
+ break;
+ }
+
+ if (out_of_frame)
+ {
+ *result = false;
+ return -1;
+ }
+ }
+ return current_pos;
+}
+
+/*
+ * get_slots
+ * Get current, previous and next tuples.
+ * Returns false if current row is out of partition/full frame.
+ */
+static
+bool
+get_slots(WindowObject winobj, int64 current_pos)
+{
+ WindowAggState *winstate = winobj->winstate;
+ TupleTableSlot *slot;
+ int ret;
+ ExprContext *econtext;
+
+ econtext = winstate->ss.ps.ps_ExprContext;
+
+ /* set up current row tuple slot */
+ slot = winstate->temp_slot_1;
+ if (!window_gettupleslot(winobj, current_pos, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of partition at:" INT64_FORMAT,
+ current_pos);
+#endif
+ return false;
+ }
+ ret = row_is_in_frame(winstate, current_pos, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "current row is out of frame at: " INT64_FORMAT,
+ current_pos);
+#endif
+ ExecClearTuple(slot);
+ return false;
+ }
+ econtext->ecxt_outertuple = slot;
+
+ /* for PREV */
+ if (current_pos > 0)
+ {
+ slot = winstate->prev_slot;
+ if (!window_gettupleslot(winobj, current_pos - 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of partition at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos - 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "previous row is out of frame at: " INT64_FORMAT,
+ current_pos - 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_scantuple = winstate->null_slot;
+ }
+ else
+ {
+ econtext->ecxt_scantuple = slot;
+ }
+ }
+ }
+ else
+ econtext->ecxt_scantuple = winstate->null_slot;
+
+ /* for NEXT */
+ slot = winstate->next_slot;
+ if (!window_gettupleslot(winobj, current_pos + 1, slot))
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of partiton at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ {
+ ret = row_is_in_frame(winstate, current_pos + 1, slot);
+ if (ret <= 0)
+ {
+#ifdef RPR_DEBUG
+ elog(DEBUG1, "next row is out of frame at: " INT64_FORMAT,
+ current_pos + 1);
+#endif
+ ExecClearTuple(slot);
+ econtext->ecxt_innertuple = winstate->null_slot;
+ }
+ else
+ econtext->ecxt_innertuple = slot;
+ }
+ return true;
+}
+
+/*
+ * pattern_initial
+ * Return pattern variable initial character
+ * matching with pattern variable name vname.
+ * If not found, return 0.
+ */
+static
+char
+pattern_initial(WindowAggState *winstate, char *vname)
+{
+ char initial;
+ char *name;
+ ListCell *lc1,
+ *lc2;
+
+ forboth(lc1, winstate->defineVariableList,
+ lc2, winstate->defineInitial)
+ {
+ name = strVal(lfirst(lc1)); /* DEFINE variable name */
+ initial = *(strVal(lfirst(lc2))); /* DEFINE variable initial */
+
+
+ if (!strcmp(name, vname))
+ return initial; /* found */
+ }
+ return 0;
+}
+
+/*
+ * string_set_init
+ * Create dynamic set of StringInfo.
+ */
+static
+StringSet * string_set_init(void)
+{
+/* Initial allocation size of str_set */
+#define STRING_SET_ALLOC_SIZE 1024
+
+ StringSet *string_set;
+ Size set_size;
+
+ string_set = palloc0(sizeof(StringSet));
+ string_set->set_index = 0;
+ set_size = STRING_SET_ALLOC_SIZE;
+ string_set->str_set = palloc(set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+
+ return string_set;
+}
+
+/*
+ * string_set_add
+ * Add StringInfo str to StringSet string_set.
+ */
+static
+void
+string_set_add(StringSet * string_set, StringInfo str)
+{
+ Size set_size;
+
+ set_size = string_set->set_size;
+ if (string_set->set_index >= set_size)
+ {
+ set_size *= 2;
+ string_set->str_set = repalloc(string_set->str_set,
+ set_size * sizeof(StringInfo));
+ string_set->set_size = set_size;
+ }
+
+ string_set->str_set[string_set->set_index++] = str;
+
+ return;
+}
+
+/*
+ * string_set_get
+ * Returns StringInfo specified by index.
+ * If there's no data yet, returns NULL.
+ */
+static
+StringInfo
+string_set_get(StringSet * string_set, int index)
+{
+ /* no data? */
+ if (index == 0 && string_set->set_index == 0)
+ return NULL;
+
+ if (index < 0 || index >= string_set->set_index)
+ elog(ERROR, "invalid index: %d", index);
+
+ return string_set->str_set[index];
+}
+
+/*
+ * string_set_get_size
+ * Returns the size of StringSet.
+ */
+static
+int
+string_set_get_size(StringSet * string_set)
+{
+ return string_set->set_index;
+}
+
+/*
+ * string_set_discard
+ * Discard StringSet.
+ * All memory including StringSet itself is freed.
+ */
+static
+void
+string_set_discard(StringSet * string_set)
+{
+ int i;
+
+ for (i = 0; i < string_set->set_index; i++)
+ {
+ StringInfo str = string_set->str_set[i];
+
+ if (str)
+ {
+ pfree(str->data);
+ pfree(str);
+ }
+ }
+ pfree(string_set->str_set);
+ pfree(string_set);
+}
+
+/*
+ * variable_pos_init
+ * Create and initialize variable postion structure
+ */
+static
+VariablePos * variable_pos_init(void)
+{
+ VariablePos *variable_pos;
+
+ variable_pos = palloc(sizeof(VariablePos) * NUM_ALPHABETS);
+ MemSet(variable_pos, -1, sizeof(VariablePos) * NUM_ALPHABETS);
+ return variable_pos;
+}
+
+/*
+ * variable_pos_register
+ * Register pattern variable whose initial is initial into postion index.
+ * pos is position of initial.
+ * If pos is already registered, register it at next empty slot.
+ */
+static
+void
+variable_pos_register(VariablePos * variable_pos, char initial, int pos)
+{
+ int index = initial - 'a';
+ int slot;
+ int i;
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ for (i = 0; i < NUM_ALPHABETS; i++)
+ {
+ slot = variable_pos[index].pos[i];
+ if (slot < 0)
+ {
+ /* empty slot found */
+ variable_pos[index].pos[i] = pos;
+ return;
+ }
+ }
+ elog(ERROR, "no empty slot for initial: %c", initial);
+}
+
+/*
+ * variable_pos_compare
+ * Returns true if initial1 can be followed by initial2
+ */
+static
+bool
+variable_pos_compare(VariablePos * variable_pos, char initial1, char initial2)
+{
+ int index1,
+ index2;
+ int pos1,
+ pos2;
+
+ for (index1 = 0;; index1++)
+ {
+ pos1 = variable_pos_fetch(variable_pos, initial1, index1);
+ if (pos1 < 0)
+ break;
+
+ for (index2 = 0;; index2++)
+ {
+ pos2 = variable_pos_fetch(variable_pos, initial2, index2);
+ if (pos2 < 0)
+ break;
+ if (pos1 <= pos2)
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * variable_pos_fetch
+ * Fetch position of pattern variable whose initial is initial, and whose index
+ * is index. If no postion was registered by initial, index, returns -1.
+ */
+static
+int
+variable_pos_fetch(VariablePos * variable_pos, char initial, int index)
+{
+ int pos = initial - 'a';
+
+ if (pos < 0 || pos > NUM_ALPHABETS)
+ elog(ERROR, "initial is not valid char: %c", initial);
+
+ if (index < 0 || index > NUM_ALPHABETS)
+ elog(ERROR, "index is not valid: %d", index);
+
+ return variable_pos[pos].pos[index];
+}
+
+/*
+ * variable_pos_discard
+ * Discard VariablePos
+ */
+static
+void
+variable_pos_discard(VariablePos * variable_pos)
+{
+ pfree(variable_pos);
+}
diff --git a/src/backend/utils/adt/windowfuncs.c b/src/backend/utils/adt/windowfuncs.c
index 473c61569f..92c528d38c 100644
--- a/src/backend/utils/adt/windowfuncs.c
+++ b/src/backend/utils/adt/windowfuncs.c
@@ -13,6 +13,9 @@
*/
#include "postgres.h"
+#include "catalog/pg_collation_d.h"
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
#include "nodes/parsenodes.h"
#include "nodes/supportnodes.h"
#include "utils/fmgrprotos.h"
@@ -37,11 +40,19 @@ typedef struct
int64 remainder; /* (total rows) % (bucket num) */
} ntile_context;
+/*
+ * rpr process information.
+ * Used for AFTER MATCH SKIP PAST LAST ROW
+ */
+typedef struct SkipContext
+{
+ int64 pos; /* last row absolute position */
+} SkipContext;
+
static bool rank_up(WindowObject winobj);
static Datum leadlag_common(FunctionCallInfo fcinfo,
bool forward, bool withoffset, bool withdefault);
-
/*
* utility routine for *_rank functions.
*/
@@ -674,7 +685,7 @@ window_last_value(PG_FUNCTION_ARGS)
bool isnull;
result = WinGetFuncArgInFrame(winobj, 0,
- 0, WINDOW_SEEK_TAIL, true,
+ 0, WINDOW_SEEK_TAIL, false,
&isnull, NULL);
if (isnull)
PG_RETURN_NULL();
@@ -714,3 +725,25 @@ window_nth_value(PG_FUNCTION_ARGS)
PG_RETURN_DATUM(result);
}
+
+/*
+ * prev
+ * Dummy function to invoke RPR's navigation operator "PREV".
+ * This is *not* a window function.
+ */
+Datum
+window_prev(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
+
+/*
+ * next
+ * Dummy function to invoke RPR's navigation operation "NEXT".
+ * This is *not* a window function.
+ */
+Datum
+window_next(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_DATUM(PG_GETARG_DATUM(0));
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 43f608d7a0..6a301920f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10604,6 +10604,12 @@
{ oid => '3114', descr => 'fetch the Nth row value',
proname => 'nth_value', prokind => 'w', prorettype => 'anyelement',
proargtypes => 'anyelement int4', prosrc => 'window_nth_value' },
+{ oid => '6122', descr => 'previous value',
+ proname => 'prev', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_prev' },
+{ oid => '6123', descr => 'next value',
+ proname => 'next', provolatile => 's', prorettype => 'anyelement',
+ proargtypes => 'anyelement', prosrc => 'window_next' },
# functions for range types
{ oid => '3832', descr => 'I/O',
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 88467977f8..058cef2121 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -2581,6 +2581,11 @@ typedef enum WindowAggStatus
* tuples during spool */
} WindowAggStatus;
+#define RF_NOT_DETERMINED 0
+#define RF_FRAME_HEAD 1
+#define RF_SKIPPED 2
+#define RF_UNMATCHED 3
+
typedef struct WindowAggState
{
ScanState ss; /* its first field is NodeTag */
@@ -2640,6 +2645,19 @@ typedef struct WindowAggState
int64 groupheadpos; /* current row's peer group head position */
int64 grouptailpos; /* " " " " tail position (group end+1) */
+ /* these fields are used in Row pattern recognition: */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ List *patternVariableList; /* list of row pattern variables names
+ * (list of String) */
+ List *patternRegexpList; /* list of row pattern regular expressions
+ * ('+' or ''. list of String) */
+ List *defineVariableList; /* list of row pattern definition
+ * variables (list of String) */
+ List *defineClauseList; /* expression for row pattern definition
+ * search conditions ExprState list */
+ List *defineInitial; /* list of row pattern definition variable
+ * initials (list of String) */
+
MemoryContext partcontext; /* context for partition-lifespan data */
MemoryContext aggcontext; /* shared context for aggregate working data */
MemoryContext curaggcontext; /* current aggregate's working data */
@@ -2667,6 +2685,18 @@ typedef struct WindowAggState
TupleTableSlot *agg_row_slot;
TupleTableSlot *temp_slot_1;
TupleTableSlot *temp_slot_2;
+
+ /* temporary slots for RPR */
+ TupleTableSlot *prev_slot; /* PREV row navigation operator */
+ TupleTableSlot *next_slot; /* NEXT row navigation operator */
+ TupleTableSlot *null_slot; /* all NULL slot */
+
+ /*
+ * Each byte corresponds to a row positioned at absolute its pos in
+ * partition. See above definition for RF_*
+ */
+ char *reduced_frame_map;
+ int64 alloc_sz; /* size of the map */
} WindowAggState;
/* ----------------
--
2.25.1
----Next_Part(Thu_Sep_19_13_59_47_2024_608)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v22-0006-Row-pattern-recognition-patch-docs.patch"
^ permalink raw reply [nested|flat] 117+ messages in thread
end of thread, other threads:[~2024-09-19 04:48 UTC | newest]
Thread overview: 117+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-12-25 21:23 POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Alex Shulgin <[email protected]>
2014-12-26 10:41 ` Pavel Stehule <[email protected]>
2014-12-26 10:49 ` Pavel Stehule <[email protected]>
2021-12-18 08:55 ` Damir Belyalov <[email protected]>
2021-12-19 05:09 ` Pavel Stehule <[email protected]>
2022-07-19 12:40 ` Damir Belyalov <[email protected]>
2022-08-15 12:29 ` torikoshia <[email protected]>
2022-08-15 13:23 ` Damir Belyalov <[email protected]>
2022-08-22 12:46 ` torikoshia <[email protected]>
2022-08-24 16:54 ` Damir Belyalov <[email protected]>
2022-08-24 16:57 ` Damir Belyalov <[email protected]>
2022-08-29 13:02 ` torikoshia <[email protected]>
2022-09-21 12:11 ` Damir Belyalov <[email protected]>
2022-09-26 13:04 ` torikoshia <[email protected]>
2022-09-29 13:18 ` Damir Belyalov <[email protected]>
2022-10-17 10:17 ` Damir Belyalov <[email protected]>
2022-11-02 08:46 ` Damir Belyalov <[email protected]>
2022-12-07 11:15 ` Nikita Malakhov <[email protected]>
2022-12-09 14:25 ` Danil Anisimow <[email protected]>
2020-03-31 19:40 [PATCH v15 10/10] pg_ls_* to show file type and show special files Justin Pryzby <[email protected]>
2022-08-24 22:47 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Zhihong Yu <[email protected]>
2023-06-25 11:48 [PATCH v1 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-06-26 08:05 [PATCH v2 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-07-26 10:49 [PATCH v3 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-08-09 07:56 [PATCH v4 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-09-02 06:32 [PATCH v5 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-09-12 05:22 [PATCH v6 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-09-15 10:02 Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features) Damir Belyalov <[email protected]>
2023-09-19 14:00 ` torikoshia <[email protected]>
2023-09-20 16:15 ` Damir <[email protected]>
2023-11-08 18:18 ` Tom Lane <[email protected]>
2023-11-08 19:34 ` Daniel Gustafsson <[email protected]>
2023-11-08 20:12 ` Tom Lane <[email protected]>
2023-11-09 04:33 ` [email protected]
2023-11-14 10:10 ` Damir Belyalov <[email protected]>
2023-11-14 10:16 ` Alena Rybakina <[email protected]>
2023-11-15 01:23 ` [email protected]
2023-11-24 03:52 ` Andrei Lepikhov <[email protected]>
2023-12-04 02:23 ` jian he <[email protected]>
2023-12-05 10:07 ` Alena Rybakina <[email protected]>
2023-12-06 10:47 ` jian he <[email protected]>
2023-12-08 07:09 ` Alena Rybakina <[email protected]>
2023-12-10 10:32 ` jian he <[email protected]>
2023-12-11 14:05 ` Alena Rybakina <[email protected]>
2023-12-12 13:04 ` jian he <[email protected]>
2023-12-14 14:48 ` Alena Rybakina <[email protected]>
2023-12-14 20:48 ` Masahiko Sawada <[email protected]>
2023-12-18 00:15 ` jian he <[email protected]>
2023-12-18 05:09 ` torikoshia <[email protected]>
2023-12-18 07:41 ` jian he <[email protected]>
2023-12-19 01:13 ` Masahiko Sawada <[email protected]>
2023-12-20 04:07 ` jian he <[email protected]>
2023-12-20 12:26 ` Masahiko Sawada <[email protected]>
2023-12-28 03:57 ` jian he <[email protected]>
2024-01-04 16:05 ` vignesh C <[email protected]>
2024-01-05 08:37 ` jian he <[email protected]>
2024-01-06 00:50 ` jian he <[email protected]>
2024-01-09 14:36 ` torikoshia <[email protected]>
2024-01-10 07:42 ` Masahiko Sawada <[email protected]>
2024-01-11 03:13 ` jian he <[email protected]>
2024-01-12 02:58 ` torikoshia <[email protected]>
2024-01-13 14:19 ` jian he <[email protected]>
2024-01-14 01:30 ` Alexander Korotkov <[email protected]>
2024-01-14 20:34 ` Masahiko Sawada <[email protected]>
2024-01-14 23:21 ` Alexander Korotkov <[email protected]>
2024-01-15 06:43 ` Masahiko Sawada <[email protected]>
2024-01-15 15:17 ` Alexander Korotkov <[email protected]>
2024-01-16 00:27 ` torikoshia <[email protected]>
2024-01-16 15:08 ` Alexander Korotkov <[email protected]>
2024-01-17 05:38 ` torikoshia <[email protected]>
2024-01-17 07:48 ` Kyotaro Horiguchi <[email protected]>
2024-01-17 21:06 ` Alexander Korotkov <[email protected]>
2024-01-17 21:37 ` Tom Lane <[email protected]>
2024-01-18 00:56 ` Masahiko Sawada <[email protected]>
2024-01-18 01:10 ` jian he <[email protected]>
2024-01-18 02:15 ` torikoshia <[email protected]>
2024-01-18 07:59 ` Alexander Korotkov <[email protected]>
2024-01-18 08:01 ` Pavel Stehule <[email protected]>
2024-01-18 08:33 ` Masahiko Sawada <[email protected]>
2024-03-28 01:20 ` Masahiko Sawada <[email protected]>
2024-03-28 12:38 ` torikoshia <[email protected]>
2024-03-28 12:54 ` Masahiko Sawada <[email protected]>
2024-03-29 02:54 ` torikoshia <[email protected]>
2024-04-01 02:31 ` Masahiko Sawada <[email protected]>
2024-04-02 10:34 ` torikoshia <[email protected]>
2024-04-16 04:16 ` Masahiko Sawada <[email protected]>
2024-04-17 07:28 ` torikoshia <[email protected]>
2024-04-17 07:30 ` Masahiko Sawada <[email protected]>
2024-01-18 12:09 ` torikoshia <[email protected]>
2024-01-18 14:59 ` jian he <[email protected]>
2024-01-19 12:37 ` torikoshia <[email protected]>
2024-01-19 13:27 ` Alexander Korotkov <[email protected]>
2024-01-19 14:26 ` torikoshia <[email protected]>
2024-01-17 21:01 ` Alexander Korotkov <[email protected]>
2023-12-19 00:28 ` Masahiko Sawada <[email protected]>
2023-12-18 02:41 ` torikoshia <[email protected]>
2023-11-16 00:00 ` jian he <[email protected]>
2023-11-08 23:53 ` Andres Freund <[email protected]>
2023-11-09 00:00 ` Tom Lane <[email protected]>
2023-11-09 00:26 ` Andres Freund <[email protected]>
2023-11-09 00:28 ` Damir Belyalov <[email protected]>
2023-09-22 04:53 [PATCH v7 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-09-25 05:01 [PATCH v8 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-10-04 05:51 [PATCH v9 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-10-22 02:22 [PATCH v10 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-11-08 06:57 [PATCH v11 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2023-12-04 11:23 [PATCH v12 4/7] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-01-22 09:45 [PATCH v13 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-02-28 13:59 [PATCH v14 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-03-28 10:30 [PATCH v15 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-04-12 06:49 [PATCH v16 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-04-28 11:00 [PATCH v17 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-05-11 07:11 [PATCH v18 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-05-14 23:26 [PATCH v19 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-05-24 02:26 [PATCH v20 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-08-26 04:32 [PATCH v21 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
2024-09-19 04:48 [PATCH v22 5/8] Row pattern recognition patch (executor). Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox