pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: COPY FROM performance improvements
52+ messages / 10 participants
[nested] [flat]
* Re: COPY FROM performance improvements
@ 2005-06-27 06:23 Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-06-27 06:23 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@surnet.cl>; +Cc: Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Alvaro,
> Am I the only one annoyed by the fact that the patch is not very nice to
> 80-columns-wide terminals? It doesn't need to be a rigid rule but I
> think the code looks much better if it's not too wide. This code is
> wide already, but I think we should be making it better, not the other
> way around.
Yup - fixed (as well as I can without mucking readability).
> Also, your text editor seems to be messing the indentation of comments
> when there are ( or other symbols in the comment text. (Maybe this
> doesn't matter a lot because pgindent will fix it, but still -- it makes
> it slightly more difficult to read.)
Yah - I think I fixed several mis-indented comments. I'm using vim with
tabstop=4. I personally don't like tabs in text and would prefer them
expanded using spaces, but that's a nice way to make small formatting
changes look huge in a cvs diff.
See attached - only formatting changes included.
- Luke
Attachments:
fast_copy_V11_8.1.patch (application/octet-stream, 56.3K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.245
diff -c -r1.245 copy.c
*** src/backend/commands/copy.c 2 Jun 2005 01:21:22 -0000 1.245
--- src/backend/commands/copy.c 27 Jun 2005 06:19:56 -0000
***************
*** 51,56 ****
--- 51,57 ----
#define ISOCTAL(c) (((c) >= '0') && ((c) <= '7'))
#define OCTVALUE(c) ((c) - '0')
+ #define COPY_BUF_SIZE 65536
/*
* Represents the different source/dest cases we need to worry about at
***************
*** 84,90 ****
EOL_CRNL
} EolType;
-
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/*
--- 85,90 ----
***************
*** 92,97 ****
--- 92,100 ----
* never been reentrant...
*/
static CopyDest copy_dest;
+ static int eol_ch[2]; /* The byte values of the 1 or 2 eol bytes */
+ static char escapec; /* escape char for delimited data format. */
+ static bool client_encoding_only; /* true if client encoding is a non supported server encoding */
static FILE *copy_file; /* used if copy_dest == COPY_FILE */
static StringInfo copy_msgbuf; /* used if copy_dest == COPY_NEW_FE */
static bool fe_eof; /* true if detected end of copy data */
***************
*** 101,131 ****
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname; /* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname; /* current att for error messages */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attribute_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attribute_buf's data buffer!
*/
! static StringInfoData attribute_buf;
!
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attribute_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf;
static bool line_buf_converted;
/* non-export function prototypes */
--- 104,144 ----
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname;/* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname;/* current att for error messages */
!
! /* Static variables for buffered input parsing */
! /* leave room for extra null (to enable use of string functions) */
! static char input_buf[COPY_BUF_SIZE + 1];
! static char *begloc;
! static char *endloc;
! static bool esc_in_prevbuf; /* backslash was last character of the data input buffer */
! static bool cr_in_prevbuf; /* CR was last character of the data input buffer */
! static bool line_done; /* finished processing the whole line or stopped in the middle */
! static bool buf_done; /* finished processing the current buffer */
! static int buffer_index; /* input buffer index */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attr_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attr_buf's data buffer!
*/
! static StringInfoData attr_buf; /* still used in CopyFrom() */
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attr_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf; /* still used in CopyFromBinary/CSV() */
static bool line_buf_converted;
/* non-export function prototypes */
***************
*** 151,156 ****
--- 164,182 ----
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
+ static bool CopyReadLineBuffered(size_t bytesread);
+ static void CopyFromDelimited(Relation rel, List *attnumlist, bool binary, bool oids,
+ char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
+ List *force_notnull_atts);
+ static char *CopyReadAttribute(const char *delim, const char *null_print,
+ CopyReadResult *result, bool *isnull);
+ static void CopyReadAllAttrs(const char *delim, const char *null_print, int null_print_len,
+ char *nulls, List *attnumlist , int *attr_offsets, int num_phys_attrs,
+ Form_pg_attribute *attr);
+ static char *CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
+ CopyReadResult *result, bool *isnull);
+ static bool DetectLineEnd(size_t bytesread);
+
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
***************
*** 160,166 ****
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static void CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
--- 186,192 ----
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static int CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
***************
*** 171,176 ****
--- 197,205 ----
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
+ /* new parsing utils */
+ static char *strchrlen(const char *s, int c, size_t len);
+ static char *str2chrlen(const char *s, int c1, int c2, size_t len, char *c_found);
/*
* Send copy start/stop messages for frontend copies. These have changed
***************
*** 383,396 ****
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
*/
! static void
CopyGetData(void *databuf, int datasize)
{
switch (copy_dest)
{
case COPY_FILE:
! fread(databuf, datasize, 1, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
--- 412,430 ----
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
+ *
+ * Returns: the number of bytes that were successfully read
+ * into the data buffer.
*/
! static int
CopyGetData(void *databuf, int datasize)
{
+ size_t bytesread = 0;
+
switch (copy_dest)
{
case COPY_FILE:
! bytesread = fread(databuf, 1, datasize, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
***************
*** 402,407 ****
--- 436,442 ----
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
+ bytesread += datasize; /* update the count of bytes that were read so far */
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
***************
*** 413,419 ****
/* Try to receive another message */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
--- 448,454 ----
/* Try to receive another message */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
***************
*** 430,436 ****
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
--- 465,471 ----
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return bytesread;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
***************
*** 460,469 ****
--- 495,507 ----
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
+ bytesread += avail; /* update the count of bytes that were read so far */
datasize -= avail;
}
break;
}
+
+ return bytesread;
}
static int
***************
*** 709,715 ****
AclResult aclresult;
/* Extract options from the statement node tree */
! foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 747,753 ----
AclResult aclresult;
/* Extract options from the statement node tree */
! foreach (option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
***************
*** 828,833 ****
--- 866,873 ----
escape = quote;
}
+ escapec = '\\'; /* will be configurable in the future */
+
/* Only single-character delimiter strings are supported. */
if (strlen(delim) != 1)
ereport(ERROR,
***************
*** 935,941 ****
force_quote_atts = CopyGetAttnums(rel, force_quote);
! foreach(cur, force_quote_atts)
{
int attnum = lfirst_int(cur);
--- 975,981 ----
force_quote_atts = CopyGetAttnums(rel, force_quote);
! foreach (cur, force_quote_atts)
{
int attnum = lfirst_int(cur);
***************
*** 956,962 ****
force_notnull_atts = CopyGetAttnums(rel, force_notnull);
! foreach(cur, force_notnull_atts)
{
int attnum = lfirst_int(cur);
--- 996,1002 ----
force_notnull_atts = CopyGetAttnums(rel, force_notnull);
! foreach (cur, force_notnull_atts)
{
int attnum = lfirst_int(cur);
***************
*** 969,981 ****
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
--- 1009,1039 ----
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attr_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
+ /*
+ * check if the client encoding is one of the 5 encodings
+ * that are not supported as a server encodings.
+ */
+ switch(client_encoding)
+ {
+ case PG_SJIS:
+ case PG_BIG5:
+ case PG_GBK:
+ case PG_UHC:
+ case PG_GB18030:
+ client_encoding_only = true;
+ break;
+ default:
+ client_encoding_only = false;
+ }
+
+
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
***************
*** 1029,1036 ****
--- 1087,1101 ----
errmsg("\"%s\" is a directory", filename)));
}
}
+
+ if (csv_mode || binary) /* old path */
CopyFrom(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
quote, escape, force_notnull_atts, header_line);
+ /* new path for improved performance (only for delimited format for now) */
+ else
+ CopyFromDelimited(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
+ quote, escape, force_notnull_atts);
+
}
else
{ /* copy from database to file */
***************
*** 1106,1114 ****
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attribute_buf.data);
pfree(line_buf.data);
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
--- 1171,1180 ----
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attr_buf.data);
pfree(line_buf.data);
+
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
***************
*** 1181,1187 ****
/* Get info about the columns we need to process. */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
! foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
Oid out_func_oid;
--- 1247,1253 ----
/* Get info about the columns we need to process. */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
! foreach (cur, attnumlist)
{
int attnum = lfirst_int(cur);
Oid out_func_oid;
***************
*** 1248,1254 ****
bool hdr_delim = false;
char *colname;
! foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 1314,1320 ----
bool hdr_delim = false;
char *colname;
! foreach (cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1304,1310 ****
}
}
! foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
Datum value;
--- 1370,1376 ----
}
}
! foreach (cur, attnumlist)
{
int attnum = lfirst_int(cur);
Datum value;
***************
*** 1346,1353 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
--- 1412,1420 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
***************
*** 1396,1405 ****
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attribute_buf.data);
}
else
{
--- 1463,1472 ----
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attr_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attr_buf.data);
}
else
{
***************
*** 1407,1412 ****
--- 1474,1481 ----
if (line_buf_converted ||
client_encoding == server_encoding)
{
+ /* Strip off the newline */
+ *(line_buf.data + line_buf.len - 1) ='\0';
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
copy_relname, copy_lineno,
***************
*** 1460,1465 ****
--- 1529,1535 ----
appendStringInfoString(buf, "...");
}
+
/*
* Copy FROM file to relation.
*/
***************
*** 1761,1767 ****
}
/* Loop to read the user attributes on the line. */
! foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
--- 1831,1837 ----
}
/* Loop to read the user attributes on the line. */
! foreach (cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
***************
*** 1855,1861 ****
}
i = 0;
! foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
--- 1925,1931 ----
}
i = 0;
! foreach (cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
***************
*** 1987,1992 ****
--- 2057,2506 ----
FreeExecutorState(estate);
}
+ /*
+ * Copy FROM file to relation with faster processing.
+ */
+ static void
+ CopyFromDelimited(Relation rel, List *attnumlist, bool binary, bool oids,
+ char *delim, char *null_print, bool csv_mode, char *quote,
+ char *escape, List *force_notnull_atts)
+ {
+ HeapTuple tuple;
+ TupleDesc tupDesc;
+ Form_pg_attribute *attr;
+ AttrNumber num_phys_attrs,
+ attr_count,
+ num_defaults;
+ FmgrInfo *in_functions;
+ Oid *typioparams;
+ ExprState **constraintexprs;
+ bool *force_notnull;
+ bool hasConstraints = false;
+ int attnum;
+ int i;
+ Oid in_func_oid;
+ Datum *values;
+ char *nulls;
+ /* an array of offsets into attr_buf that point to beginning of attributes */
+ int *attr_offsets;
+ bool isnull;
+ int null_print_len; /* length of null print */
+ ResultRelInfo *resultRelInfo;
+ EState *estate = CreateExecutorState(); /* for ExecConstraints() */
+ TupleTableSlot *slot;
+ bool file_has_oids;
+ int *defmap;
+ ExprState **defexprs; /* array of default att expressions */
+ ExprContext *econtext; /* used for ExecEvalExpr for default atts */
+ MemoryContext oldcontext = CurrentMemoryContext;
+ ErrorContextCallback errcontext;
+ bool no_more_data;
+ CopyReadResult result;
+ ListCell *cur;
+
+
+ tupDesc = RelationGetDescr(rel);
+ attr = tupDesc->attrs;
+ num_phys_attrs = tupDesc->natts;
+ attr_count = list_length(attnumlist);
+ num_defaults = 0;
+
+ /*
+ * We need a ResultRelInfo so we can use the regular executor's
+ * index-entry-making machinery. (There used to be a huge amount of
+ * code here that basically duplicated execUtils.c ...)
+ */
+ resultRelInfo = makeNode(ResultRelInfo);
+ resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
+ resultRelInfo->ri_RelationDesc = rel;
+ resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
+ if (resultRelInfo->ri_TrigDesc)
+ resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
+ palloc0(resultRelInfo->ri_TrigDesc->numtriggers * sizeof(FmgrInfo));
+ resultRelInfo->ri_TrigInstrument = NULL;
+
+ ExecOpenIndices(resultRelInfo);
+
+ estate->es_result_relations = resultRelInfo;
+ estate->es_num_result_relations = 1;
+ estate->es_result_relation_info = resultRelInfo;
+
+ /* Set up a tuple slot too */
+ slot = MakeSingleTupleTableSlot(tupDesc);
+
+ econtext = GetPerTupleExprContext(estate);
+
+ /*
+ * Pick up the required catalog information for each attribute in the
+ * relation, including the input function, the element type (to pass
+ * to the input function), and info about defaults and constraints.
+ * (Which input function we use depends on text/binary format choice.)
+ */
+ in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
+ typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
+ defmap = (int *) palloc(num_phys_attrs * sizeof(int));
+ defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
+ constraintexprs = (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
+ force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
+
+ for (attnum = 1; attnum <= num_phys_attrs; attnum++)
+ {
+ /* We don't need info for dropped attributes */
+ if (attr[attnum - 1]->attisdropped)
+ continue;
+
+ getTypeInputInfo(attr[attnum - 1]->atttypid,
+ &in_func_oid, &typioparams[attnum - 1]);
+
+ fmgr_info(in_func_oid, &in_functions[attnum - 1]);
+
+ if (list_member_int(force_notnull_atts, attnum))
+ force_notnull[attnum - 1] = true;
+ else
+ force_notnull[attnum - 1] = false;
+
+ /* Get default info if needed */
+ if (!list_member_int(attnumlist, attnum))
+ {
+ /* attribute is NOT to be copied from input */
+ /* use default value if one exists */
+ Node *defexpr = build_column_default(rel, attnum);
+
+ if (defexpr != NULL)
+ {
+ defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
+ estate);
+ defmap[num_defaults] = attnum - 1;
+ num_defaults++;
+ }
+ }
+
+ /* If it's a domain type, set up to check domain constraints */
+ if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
+ {
+ Param *prm;
+ Node *node;
+
+ /*
+ * Easiest way to do this is to use parse_coerce.c to set up
+ * an expression that checks the constraints. (At present,
+ * the expression might contain a length-coercion-function
+ * call and/or CoerceToDomain nodes.) The bottom of the
+ * expression is a Param node so that we can fill in the
+ * actual datum during the data input loop.
+ */
+ prm = makeNode(Param);
+ prm->paramkind = PARAM_EXEC;
+ prm->paramid = 0;
+ prm->paramtype = getBaseType(attr[attnum - 1]->atttypid);
+
+ node = coerce_to_domain((Node *) prm,
+ prm->paramtype,
+ attr[attnum - 1]->atttypid,
+ COERCE_IMPLICIT_CAST, false, false);
+
+ constraintexprs[attnum - 1] = ExecPrepareExpr((Expr *) node, estate);
+ hasConstraints = true;
+ }
+ }
+
+ /*
+ * Prepare to catch AFTER triggers.
+ */
+ AfterTriggerBeginQuery();
+
+ /*
+ * Check BEFORE STATEMENT insertion triggers. It's debateable whether
+ * we should do this for COPY, since it's not really an "INSERT"
+ * statement as such. However, executing these triggers maintains
+ * consistency with the EACH ROW triggers that we already fire on
+ * COPY.
+ */
+ ExecBSInsertTriggers(estate, resultRelInfo);
+
+ file_has_oids = oids; /* must rely on user to tell us this... */
+
+ values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
+ nulls = (char *) palloc(num_phys_attrs * sizeof(char));
+ attr_offsets = (int *) palloc(num_phys_attrs * sizeof(int));
+
+ /* Make room for a PARAM_EXEC value for domain constraint checks */
+ if (hasConstraints)
+ econtext->ecxt_param_exec_vals = (ParamExecData *)
+ palloc0(sizeof(ParamExecData));
+
+ /* Initialize static variables */
+ fe_eof = false;
+ eol_type = EOL_UNKNOWN;
+ copy_binary = binary;
+ copy_relname = RelationGetRelationName(rel);
+ copy_lineno = 0;
+ copy_attname = NULL;
+ line_buf_converted = false;
+
+ /* it is MUCH faster to do this once here than in the attribute parse loop */
+ null_print_len = strlen(null_print);
+
+ /* Set up callback to identify error line number */
+ errcontext.callback = copy_in_error_callback;
+ errcontext.arg = NULL;
+ errcontext.previous = error_context_stack;
+ error_context_stack = &errcontext;
+
+ /* Set up data buffer to hold a chunk of data*/
+ MemSet(input_buf, ' ', COPY_BUF_SIZE * sizeof(char));
+ input_buf[COPY_BUF_SIZE] = '\0';
+
+ no_more_data = false; /* no more input data to read from file or FE */
+ line_done = true;
+
+ do
+ {
+ /* read a chunk of data into the buffer */
+ size_t bytesread = CopyGetData(input_buf, COPY_BUF_SIZE);
+ buf_done = false;
+
+ /* set buffer pointers to beginning of the buffer */
+ begloc = input_buf;
+ buffer_index = 0;
+
+ /* continue if some bytes were read or if we didn't reach EOF. if we both *
+ * reached EOF _and_ no bytes were read, quit the loop we are done */
+ if (bytesread > 0 || !fe_eof)
+ {
+
+ while (!buf_done)
+ {
+ bool skip_tuple;
+ Oid loaded_oid = InvalidOid;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Reset the per-tuple exprcontext */
+ ResetPerTupleExprContext(estate);
+
+ /* Switch into its memory context */
+ MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+
+ /* Initialize all values for row to NULL */
+ MemSet(values, 0, num_phys_attrs * sizeof(Datum));
+ MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
+ /* reset attribute pointers */
+ MemSet(attr_offsets, 0, num_phys_attrs * sizeof(int));
+
+ result = NORMAL_ATTR;
+
+ /* Actually read the line into memory here */
+ line_done = CopyReadLineBuffered(bytesread);
+ copy_lineno++;
+
+ /* if didn't finish processing data line, skip att parsing and read more data,
+ * unless there is no more data to read... (which means that the original last
+ * data line is missing attrs and we want to catch that error)
+ */
+ if (!line_done)
+ {
+ copy_lineno--;
+ if (!fe_eof || buf_done)
+ break;
+ }
+
+ if (file_has_oids)
+ {
+ char *oid_string;
+ /* can't be in CSV mode here */
+ oid_string = CopyReadOidAttr(delim, null_print, null_print_len,
+ &result, &isnull);
+
+ if (isnull)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("null OID in COPY data")));
+ else
+ {
+ copy_attname = "oid";
+ loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
+ CStringGetDatum(oid_string)));
+ if (loaded_oid == InvalidOid)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("invalid OID in COPY data")));
+ copy_attname = NULL;
+ }
+ }
+
+ /* parse all the attribute in the data line */
+ CopyReadAllAttrs(delim, null_print, null_print_len,
+ nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
+
+ /*
+ * Loop to read the user attributes on the line.
+ */
+ foreach (cur, attnumlist)
+ {
+ int attnum = lfirst_int(cur);
+ int m = attnum - 1;
+
+ if (nulls[m] == ' ')
+ isnull = false;
+ else
+ isnull = true;
+
+ /* we read an SQL NULL, no need to do anything */
+ if (!isnull)
+ {
+ copy_attname = NameStr(attr[m]->attname);
+ values[m] = FunctionCall3(&in_functions[m],
+ CStringGetDatum(attr_buf.data + attr_offsets[m]),
+ ObjectIdGetDatum(typioparams[m]),
+ Int32GetDatum(attr[m]->atttypmod));
+ copy_attname = NULL;
+ }
+ }
+
+ /*
+ * Now compute and insert any defaults available for the columns
+ * not provided by the input data. Anything not processed here or
+ * above will remain NULL.
+ */
+ for (i = 0; i < num_defaults; i++)
+ {
+ values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext, &isnull, NULL);
+ if (!isnull)
+ nulls[defmap[i]] = ' ';
+ }
+
+ /*
+ * Next apply any domain constraints
+ */
+ if (hasConstraints)
+ {
+ ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
+
+ for (i = 0; i < num_phys_attrs; i++)
+ {
+ ExprState *exprstate = constraintexprs[i];
+
+ if (exprstate == NULL)
+ continue; /* no constraint for this attr */
+
+ /* Insert current row's value into the Param value */
+ prmdata->value = values[i];
+ prmdata->isnull = (nulls[i] == 'n');
+
+ /*
+ * Execute the constraint expression. Allow the
+ * expression to replace the value (consider e.g. a
+ * timestamp precision restriction).
+ */
+ values[i] = ExecEvalExpr(exprstate, econtext,
+ &isnull, NULL);
+ nulls[i] = isnull ? 'n' : ' ';
+ }
+ }
+
+ /*
+ * And now we can form the input tuple.
+ */
+ tuple = heap_formtuple(tupDesc, values, nulls);
+
+ if (oids && file_has_oids)
+ HeapTupleSetOid(tuple, loaded_oid);
+
+ /*
+ * Triggers and stuff need to be invoked in query context.
+ */
+ MemoryContextSwitchTo(oldcontext);
+
+ skip_tuple = false;
+
+ /* BEFORE ROW INSERT Triggers */
+ if (resultRelInfo->ri_TrigDesc &&
+ resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
+ {
+ HeapTuple newtuple;
+
+ newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
+
+ if (newtuple == NULL) /* "do nothing" */
+ skip_tuple = true;
+ else if (newtuple != tuple) /* modified by Trigger(s) */
+ {
+ heap_freetuple(tuple);
+ tuple = newtuple;
+ }
+ }
+
+ if (!skip_tuple)
+ {
+ /* Place tuple in tuple slot */
+ ExecStoreTuple(tuple, slot, InvalidBuffer, false);
+
+ /*
+ * Check the constraints of the tuple
+ */
+ if (rel->rd_att->constr)
+ ExecConstraints(resultRelInfo, slot, estate);
+
+ /*
+ * OK, store the tuple and create index entries for it
+ */
+ simple_heap_insert(rel, tuple);
+
+ if (resultRelInfo->ri_NumIndices > 0)
+ ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
+
+ /* AFTER ROW INSERT Triggers */
+ ExecARInsertTriggers(estate, resultRelInfo, tuple);
+ }
+
+ line_buf.len=0; /* we can reset line buffer now. */
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ } /* end while(!buf_done) */
+ } /* end if (bytesread > 0 || !fe_eof) */
+ else /* no bytes read, end of data */
+ {
+ no_more_data = TRUE;
+ }
+ }while(!no_more_data);
+
+ /*
+ * Done, clean up
+ */
+ error_context_stack = errcontext.previous;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ /*
+ * Execute AFTER STATEMENT insertion triggers
+ */
+ ExecASInsertTriggers(estate, resultRelInfo);
+
+ /*
+ * Handle queued AFTER triggers
+ */
+ AfterTriggerEndQuery(estate);
+
+ pfree(values);
+ pfree(nulls);
+
+ pfree(attr_offsets);
+
+ pfree(in_functions);
+ pfree(typioparams);
+ pfree(defmap);
+ pfree(defexprs);
+ pfree(constraintexprs);
+ pfree(force_notnull);
+
+ ExecDropSingleTupleTableSlot(slot);
+
+ ExecCloseIndices(resultRelInfo);
+
+ FreeExecutorState(estate);
+ }
+
/*
* Read the next input line and stash it in line_buf, with conversion to
***************
*** 2264,2269 ****
--- 2778,2784 ----
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
***************
*** 2286,2291 ****
--- 2801,3104 ----
return tolower(hex) - 'a' + 10;
}
+ /*
+ * Detected the eol type by looking at the first data row.
+ * Possible eol types are NL, CR, or CRNL. If eol type was
+ * detected, it is set and a boolean true is returned to
+ * indicated detection was successful. If the first data row
+ * is longer than the input buffer, we return false and will
+ * try again in the next buffer.
+ */
+ static bool
+ DetectLineEnd(size_t bytesread)
+ {
+ int bytes_remaining = COPY_BUF_SIZE;
+ char ch_found; /* the character found in the scan ('\n' or '\r' or escapec) */
+ char *start = input_buf;
+ char *end;
+
+ while(true)
+ {
+ bytes_remaining = COPY_BUF_SIZE - (input_buf - start);
+
+ if (bytes_remaining <= 0)
+ return false;
+
+ if ( (end = str2chrlen(start, '\n', '\r', bytes_remaining, &ch_found )) == NULL)
+ {
+ return false;
+ }
+ else
+ {
+ if (ch_found == '\n')
+ {
+ eol_type = EOL_NL;
+ eol_ch[0] = '\n';
+ eol_ch[1] = '\0';
+
+ return true;
+ }
+ if (ch_found == '\r')
+ {
+ if (*(end + 1) == '\n')
+ {
+ eol_type = EOL_CRNL;
+ eol_ch[0] = '\r';
+ eol_ch[1] = '\n';
+ }
+ else
+ {
+ eol_type = EOL_CR;
+ eol_ch[0] = '\r';
+ eol_ch[1] = '\0';
+ }
+
+ return true;
+ }
+ }
+ }
+
+ }
+
+
+ /*
+ * Finds the next data line that is in the input buffer and loads it into line_buf.
+ * Returns an indication if the line that was read is complete (if an unescaped line-end was
+ * encountered). If we reached the end of buffer before the whole line was written into the
+ * line buffer then returns false.
+ */
+ static bool
+ CopyReadLineBuffered(size_t bytesread)
+ {
+ int linesize;
+ bool transcode = (client_encoding != server_encoding);
+ char *cvt;
+ bool end_marker;
+
+ /* mark that encoding conversion hasn't occurred yet */
+ line_buf_converted = false;
+
+ /*
+ * Detect end of line type if not already detected.
+ */
+ if (eol_type == EOL_UNKNOWN)
+ {
+ bool eol_detected = DetectLineEnd(bytesread);
+
+ if (!eol_detected)
+ {
+ /* load entire input buffer into line buf, and quit */
+ appendBinaryStringInfo(&line_buf,input_buf,COPY_BUF_SIZE);
+ line_done = false;
+ buf_done = true;
+
+ return line_done;
+ }
+ }
+
+ /*
+ * Special case: eol is CRNL, last byte of previous buffer was an
+ * unescaped CR and 1st byte of current buffer is NL. We check for
+ * that here.
+ */
+ if (eol_type == EOL_CRNL)
+ {
+ /* if we started scanning from the 1st byte of the buffer */
+ if (begloc == input_buf)
+ {
+ /* and had a CR in last byte of prev buf */
+ if (cr_in_prevbuf)
+ {
+ /* if this 1st byte in buffer is 2nd byte of line end sequence (linefeed) */
+ if (*begloc == eol_ch[1])
+ {
+ /* load that one linefeed byte and indicate we are done with the data line */
+ appendBinaryStringInfo(&line_buf,begloc,1);
+ buffer_index++;
+ begloc++;
+
+ line_done = true;
+ esc_in_prevbuf = false;
+ cr_in_prevbuf = false;
+
+ return line_done;
+ }
+ }
+
+ cr_in_prevbuf = false;
+ }
+ }
+
+ /* (we need a loop so that if eol_ch is found, but prev ch is backslash, we can search
+ for the next eol_ch) */
+ while (true)
+ {
+ /* reached end of buffer */
+ if ( (endloc = strchrlen(begloc, eol_ch[0], bytesread - buffer_index)) == NULL )
+ {
+ linesize = COPY_BUF_SIZE - (begloc - input_buf);
+ appendBinaryStringInfo(&line_buf,begloc,linesize);
+
+ if (line_buf.len > 1)
+ {
+ char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
+ if ( *last_ch == escapec )
+ {
+ esc_in_prevbuf = true;
+
+ if (line_buf.len > 2)
+ {
+ last_ch--;
+ if (*last_ch == escapec)
+ esc_in_prevbuf = false;
+ }
+ }
+ else if ( *last_ch == '\r' )
+ {
+ if (eol_type == EOL_CRNL)
+ cr_in_prevbuf = true;
+ }
+ }
+
+ line_done = false;
+ buf_done = true;
+ break;
+ }
+ else /* found the 1st eol ch in input_buf. */
+ {
+ bool eol_found = true;
+ bool eol_escaped = true;
+ /*
+ * Load that piece of data (potentially a data line) into the line buffer,
+ * and update the pointers for the next scan.
+ */
+ linesize = endloc - begloc + 1;
+ appendBinaryStringInfo(&line_buf,begloc,linesize);
+ buffer_index += linesize;
+ begloc = endloc + 1;
+
+ if (eol_type == EOL_CRNL)
+ {
+ /* check if there is a '\n' after the '\r' */
+ if (*(endloc + 1) == '\n')
+ {
+ /* this is a line end */
+ appendBinaryStringInfo(&line_buf,begloc,1); /* load that '\n' */
+ buffer_index++;
+ begloc++;
+ }
+ else /* just a CR, not a line end */
+ eol_found = false;
+ }
+
+ /*
+ * in some cases, this end of line char happens to be the
+ * last character in the buffer. we need to catch that.
+ */
+ if (buffer_index >= bytesread)
+ buf_done = true;
+
+ /*
+ * Check if the 1st end of line ch is escaped.
+ */
+ if (endloc != input_buf) /* can we look 1 char back? */
+ {
+ if (*(endloc - 1) != escapec) /* prev char is not an escape */
+ eol_escaped = false;
+ else /* prev char is an escape */
+ {
+ if (endloc != (input_buf + 1)) /* can we look another char back? */
+ {
+ /* it's a double escape char, so it's not an escape */
+ if (*(endloc - 2) == escapec)
+ eol_escaped = false;
+ /* else it's a single escape char, so EOL is ascaped */
+ }
+ else
+ {
+ /* we need to check in the last buffer */
+ if (esc_in_prevbuf) /* double escape char, so not an escape */
+ eol_escaped = false;
+ }
+ }
+ }
+ else /* this eol ch is first ch in buffer, check for escape in prev buf */
+ {
+ if (!esc_in_prevbuf)
+ eol_escaped = false;
+ }
+
+ esc_in_prevbuf = false; /* reset variable */
+
+ /*
+ * if eol was found, and it isn't escaped, line is done
+ */
+ if ((eol_escaped == false) && eol_found)
+ {
+ line_done = true;
+ break;
+ }
+ else /* stay in the loop and process some more data. */
+ line_done = false;
+
+ } /* end of found eol_ch */
+ }
+
+ /*
+ * Done reading the line. Convert it to server encoding.
+ */
+ if (transcode)
+ {
+ cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
+ line_buf.len);
+ if (cvt != line_buf.data)
+ {
+ /* transfer converted data back to line_buf */
+ line_buf.len=0;
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ appendBinaryStringInfo(&line_buf,cvt,strlen(cvt));
+ }
+ }
+
+ /* indicate that conversion had occured */
+ line_buf_converted = true;
+
+ /*
+ * check if this line is an end marker -- "\."
+ */
+ end_marker = false;
+
+ switch(eol_type)
+ {
+ case EOL_NL:
+ if (!strcmp(line_buf.data,"\\.\n"))
+ end_marker = true;
+ break;
+ case EOL_CR:
+ if (!strcmp(line_buf.data,"\\.\r"))
+ end_marker = true;
+ break;
+ case EOL_CRNL:
+ if (!strcmp(line_buf.data,"\\.\r\n"))
+ end_marker = true;
+ break;
+ case EOL_UNKNOWN:
+ break;
+ }
+
+ if (end_marker)
+ {
+ fe_eof = true;
+ /* we don't want to process a \. as data line, want to quit. */
+ line_done = false;
+ buf_done = true;
+ }
+
+ return line_done;
+ }
+
+
/*----------
* Read the value of a single attribute, performing de-escaping as needed.
*
***************
*** 2314,2322 ****
int end_cursor;
int input_len;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
/* set default status */
*result = END_OF_LINE;
--- 3127,3136 ----
int end_cursor;
int input_len;
! /* reset attr_buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
/* set default status */
*result = END_OF_LINE;
***************
*** 2422,2428 ****
*/
}
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
/* check whether raw input matched null marker */
--- 3236,3242 ----
*/
}
}
! appendStringInfoCharMacro(&attr_buf, c);
}
/* check whether raw input matched null marker */
***************
*** 2433,2439 ****
else
*isnull = false;
! return attribute_buf.data;
}
--- 3247,3597 ----
else
*isnull = false;
! return attr_buf.data;
! }
!
! /*
! * Read the first attribute. This is mainly used to maintain support
! * for an OID column. All the rest of the columns will be read at once with
! * CopyReadAllAttrs().
! */
! static char *
! CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
! CopyReadResult *result, bool *isnull)
! {
! char delimc = delim[0];
! char *start_loc = line_buf.data + line_buf.cursor;
! char *end_loc;
! int attr_len = 0;
! int bytes_remaining;
!
! /* reset attribute buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! /* set default status */
! *result = END_OF_LINE;
!
! /* # of bytes that were not yet processed in this line */
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! /* got to end of line */
! if ( (end_loc = strchrlen(start_loc, delimc, bytes_remaining )) == NULL)
! {
! attr_len = bytes_remaining - 1; /* don't count '\n' in len calculation */
! appendBinaryStringInfo(&attr_buf,start_loc,attr_len);
! line_buf.cursor += attr_len + 2; /* skip '\n' and '\0' */
!
! *result = END_OF_LINE;
! }
! else /* found a delimiter */
! {
! /* (we don't care if delim was preceded with a backslash, because it's an
! invalid OID anyway) */
!
! attr_len = end_loc - start_loc; /* we don't include the delimiter ch */
!
! appendBinaryStringInfo(&attr_buf,start_loc,attr_len);
! line_buf.cursor += attr_len + 1;
!
! *result = NORMAL_ATTR;
! }
!
!
! /* check whether raw input matched null marker */
! if (attr_len == null_print_len && strncmp(start_loc, null_print, attr_len) == 0)
! *isnull = true;
! else
! *isnull = false;
!
! return attr_buf.data;
! }
!
!
! /*
! * Read all attributes. Attributes are parsed from line_buf and
! * inserted (all at once) to attr_buf, while saving pointers to
! * each attribute's starting position.
! *
! * When this routine finishes execution both the nulls array and
! * the attr_offsets array are updated. The attr_offsets will include
! * the offset from the beginning of the attribute array of which
! * each attribute begins. If a specific attribute is not used for this
! * COPY command (ommitted from the column list), a value of 0 will be assigned.
! * For example: for table foo(a,b,c,d,e) and COPY foo(a,b,e)
! * attr_offsets may look something like this after this routine
! * returns: [0,20,0,0,55]. That means that column "a" value starts
! * at byte offset 0, "b" in 20 and "e" in 55, in attr_buf.
! *
! * In the attribute buffer (attr_buf) each attribute
! * is terminated with a '\0', and therefore by using the attr_offsets
! * array we could point to a beginning of an attribute and have it
! * behave as a C string, much like previously done in COPY.
! *
! * Another aspect to improving performance is reducing the frequency
! * of data load into buffers. The original COPY read attribute code
! * loaded a character at a time. In here we try to load a chunk of data
! * at a time. Usually a chunk will include a full data row
! * (unless we have an escaped delim). That effectively reduces the number of
! * loads by a factor of number of bytes per row. This improves performance
! * greatly, unfortunately it add more complexity to the code.
! *
! * Global participants in parsing logic:
! *
! * line_buf.cursor -- an offset from beginning of the line buffer
! * that indicates where we are about to begin the next scan. Note that
! * if we have WITH OIDS this cursor is already shifted past the first
! * OID attribute.
! *
! * attr_buf.cursor -- an offset from the beginning of the
! * attribute buffer that indicates where the current attribute begins.
! */
! static void
! CopyReadAllAttrs(const char *delim, const char *null_print, int null_print_len,
! char *nulls, List *attnumlist , int *attr_offsets,
! int num_phys_attrs, Form_pg_attribute *attr)
! {
! char delimc = delim[0]; /* delimiter character */
! char *scan_start; /* pointer to line buffer where scan should start. */
! char *scan_end; /* pointer to line buffer where char was found */
! char ch_found; /* the character found in the scan (delimc or escape) */
! int attr_pre_len; /* current attr raw len, before processing escapes */
! int attr_post_len; /* current attr len after escaping */
! int m; /* attribute index being parsed */
! int bytes_remaining; /* num of bytes remaining to be scanned in line buf */
! int chunk_start; /* offset to beginning of line chunk to load */
! int chunk_len; /* length of chunk of data to load to attr buf */
! int oct_val; /* byte value for octal escapes */
! int attnum; /* attribute number being parsed */
! ListCell *cur; /* cursor to attribute list used for this COPY */
! int attribute;
!
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /* cursor is now > 0 if we copy WITH OIDS */
! scan_start = line_buf.data + line_buf.cursor;
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! attr_pre_len = 0;
! attr_post_len = 0;
!
! /*
! * Scan through the line buffer to read all attributes data
! */
! while(line_buf.cursor < line_buf.len)
! {
! bytes_remaining = line_buf.len - line_buf.cursor;
! ch_found = '\0';
!
! if ( (scan_end = str2chrlen(scan_start, delimc, escapec, bytes_remaining, &ch_found ))
! == NULL)
! {
! /* GOT TO END OF LINE BUFFER */
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* don't count eol char(s) in attr and chunk len calculation */
! if (eol_type == EOL_CRNL)
! {
! attr_pre_len += bytes_remaining - 2;
! chunk_len = line_buf.len - chunk_start - 2;
! }
! else
! {
! attr_pre_len += bytes_remaining - 1;
! chunk_len = line_buf.len - chunk_start - 1;
! }
!
! /* check if this is a NULL value or data value (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(line_buf.data + line_buf.len - attr_pre_len - 1, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! attr_offsets[m] = attr_buf.cursor;
!
!
! /* load the last chunk, the whole buffer in most cases */
! appendBinaryStringInfo(&attr_buf,line_buf.data + chunk_start,chunk_len);
!
! line_buf.cursor += attr_pre_len + 2; /* skip eol char and '\0' to exit loop */
!
! if (lnext(cur) != NULL)
! {
! /*
! * For an empty data line, the previous COPY code will
! * fail it during the conversion stage. We can fail it here
! * already, but then we will fail the regression tests b/c
! * of a different error message. that's why we return so we
! * can get the same error message that regress expects. ahh...
! * this conditional is unnecessary and should be removed soon.
! */
! if (line_buf.len > 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m + 1]->attname))));
! else
! return;
! }
! }
! else /* FOUND A DELIMITER OR ESCAPE */
! {
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! if (ch_found == delimc) /* found a delimiter */
! {
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* (we don't include the delimiter ch in length) */
! attr_pre_len += scan_end - scan_start;
! /* (we don't include the delimiter ch in length) */
! attr_post_len += scan_end - scan_start;
!
! /* check if this is a null print or data (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(scan_end - attr_pre_len, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* set the pointer to next attribute position */
! attr_offsets[m] = attr_buf.cursor;
!
! /* update buffer cursors to our current location, +1 to skip the delimc */
! line_buf.cursor = scan_end - line_buf.data + 1;
! attr_buf.cursor += attr_post_len + 1;
!
! /* prepare scan for next attr */
! scan_start = line_buf.data + line_buf.cursor;
! cur = lnext(cur);
! attr_pre_len = 0;
! attr_post_len = 0;
! }
! else /* found an escape character */
! {
! char nextc = *(scan_end + 1);
! char newc;
! int skip = 2;
!
! chunk_len = (scan_end - line_buf.data) - chunk_start + 1;
!
! /* load a chunk of data */
! appendBinaryStringInfo(&attr_buf,line_buf.data + chunk_start,chunk_len);
!
! switch(nextc)
! {
! case '0':
! case '1':
! case '2':
! case '3':
! case '4':
! case '5':
! case '6':
! case '7':
! oct_val = OCTVALUE(nextc);
! nextc = *(scan_end + 2);
! /* (no need for out bad access check since line if buffered) */
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! nextc = *(scan_end + 3);
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! }
! }
! newc = oct_val & 0377; /* the escaped byte value */
! break;
! case 'b':
! newc = '\b';
! break;
! case 'f':
! newc = '\f';
! break;
! case 'n':
! newc = '\n';
! break;
! case 'r':
! newc = '\r';
! break;
! case 't':
! newc = '\t';
! break;
! case 'v':
! newc = '\v';
! break;
! default:
! if (nextc == delimc)
! newc = delimc;
! else if (nextc == escapec)
! newc = escapec;
! else /* no escape sequence, take next char literaly */
! newc = nextc;
! break;
! }
!
! /* update to current length, add escape and escaped chars */
! attr_pre_len += scan_end - scan_start + 2;
! /* update to current length, escaped char */
! attr_post_len += scan_end - scan_start + 1;
!
! /*
! * Need to get rid of the escape character. This is done by
! * loading the chunk up to including the escape character
! * into the attribute buffer. Then overwritting the backslash
! * with the escaped sequence or char, and continuing to scan
! * from *after* the char than is after the escape in line buf.
! */
! *(attr_buf.data + attr_buf.len - 1) = newc;
! line_buf.cursor = scan_end - line_buf.data + skip;
! scan_start = scan_end + skip;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! }
!
! } /* end delimiter/backslash */
!
! } /* end line buffer scan. */
!
! /*
! * Replace all delimiters with NULL for string termination.
! * NOTE: only delimiters (NOT necessarily all delimc) are replaced.
! * Example (delimc = '|'):
! * - Before: f 1 | f \| 2 | f 3
! * - After : f 1 \0 f | 2 \0 f 3
! */
! for (attribute = 1; attribute < num_phys_attrs ; attribute++)
! {
! if (attr_offsets[attribute] != 0)
! *(attr_buf.data + attr_offsets[attribute] - 1) = '\0';
! }
!
}
***************
*** 2477,2485 ****
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
/* set default status */
*result = END_OF_LINE;
--- 3635,3644 ----
bool in_quote = false;
bool saw_quote = false;
! /* reset attr_buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
/* set default status */
*result = END_OF_LINE;
***************
*** 2519,2525 ****
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attribute_buf, nextc);
line_buf.cursor++;
continue;
}
--- 3678,3684 ----
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attr_buf, nextc);
line_buf.cursor++;
continue;
}
***************
*** 2536,2542 ****
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
if (in_quote)
--- 3695,3701 ----
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attr_buf, c);
}
if (in_quote)
***************
*** 2550,2556 ****
else
*isnull = false;
! return attribute_buf.data;
}
/*
--- 3709,3715 ----
else
*isnull = false;
! return attr_buf.data;
}
/*
***************
*** 2578,2606 ****
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! attribute_buf.cursor = 0;
! enlargeStringInfo(&attribute_buf, fld_size);
! CopyGetData(attribute_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attribute_buf.len = fld_size;
! attribute_buf.data[fld_size] = '\0';
/* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attribute_buf),
ObjectIdGetDatum(typioparam));
/* Trouble if it didn't eat the whole buffer */
! if (attribute_buf.cursor != attribute_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
--- 3737,3765 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attr_buf to empty, and load raw data in it */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! enlargeStringInfo(&attr_buf, fld_size);
! CopyGetData(attr_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attr_buf.len = fld_size;
! attr_buf.data[fld_size] = '\0';
/* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attr_buf),
ObjectIdGetDatum(typioparam));
/* Trouble if it didn't eat the whole buffer */
! if (attr_buf.cursor != attr_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
***************
*** 2777,2783 ****
/* Validate the user-supplied list and extract attnums */
ListCell *l;
! foreach(l, attnamelist)
{
char *name = strVal(lfirst(l));
int attnum;
--- 3936,3942 ----
/* Validate the user-supplied list and extract attnums */
ListCell *l;
! foreach (l, attnamelist)
{
char *name = strVal(lfirst(l));
int attnum;
***************
*** 2797,2799 ****
--- 3956,4017 ----
return attnums;
}
+
+ /******************************************************
+ * utility functions, should be placed elsewhere in src
+ ******************************************************/
+ /*
+ * These are custom versions of the string function strchr().
+ * As opposed to the original strchr which searches through
+ * a string until the target character is found, or a NULL is
+ * found, this version will not return when a NULL is found.
+ * Instead it will search through a pre-defined length of
+ * bytes and will return only if the target character(s) is reached.
+ *
+ * If our client encoding is not a supported server encoding, we
+ * know that it is not safe to look at each character as trailing
+ * byte in a multibyte character may be a 7-bit ASCII equivalent.
+ * Therefore we use pg_encoding_mblen to skip to the end of the
+ * character.
+ *
+ * parameters:
+ * s - string being searched.
+ * c(n) - char we are searching for.
+ * len - maximum # of bytes to search.
+ *
+ * returns:
+ * pointer to c - if c is located within the string.
+ * NULL - if c was not found in specified length of search. Note:
+ * this DOESN'T mean that a '\0' was reached.
+ */
+ char *strchrlen(const char *s, int c, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, s);
+
+ for (start = s; *s != c && s < start + len ; s += mblen)
+ mblen = pg_encoding_mblen(client_encoding, s);
+ }
+ else /* safe to scroll byte by byte */
+ {
+ for (start = s; *s != c && s < start + len ; s++)
+ ;
+ }
+
+ return ( (*s == c) ? (char *) s : NULL );
+ }
+
+ char *str2chrlen(const char *s, int c1, int c2, size_t len, char *c_found)
+ {
+ const char *start;
+ *c_found = '\0';
+
+ for (start = s; *s != c1 && *s != c2 && s < start + len ; s++)
+ ;
+
+ *c_found = *s;
+ return ( *c_found != '\0' ? (char *) s : NULL );
+ }
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
@ 2005-06-27 11:45 ` Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2005-06-27 11:45 UTC (permalink / raw)
To: Luke Lonergan <LLonergan@greenplum.com>; +Cc: Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Luke Lonergan wrote:
>Yah - I think I fixed several mis-indented comments. I'm using vim with
>tabstop=4. I personally don't like tabs in text and would prefer them
>expanded using spaces, but that's a nice way to make small formatting
>changes look huge in a cvs diff.
>
>
>
You might like to look at running pgindent (see src/tools/pgindent) over
the file before cutting a patch. Since this is usually run over each
file just before a release, the only badness should be things from
recent patches.
cheers
andrew
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
@ 2005-06-27 18:52 ` Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-06-27 18:52 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Andrew,
> You might like to look at running pgindent (see src/tools/pgindent) over
> the file before cutting a patch. Since this is usually run over each
> file just before a release, the only badness should be things from
> recent patches.
I've attached two patches, one gained from running pgindent against the
current CVS tip copy.c (:-D) and one gained by running the COPY FROM perf
improvements through the same. Nifty tool!
Only formatting changes in these.
- Luke
Attachments: download all (2 files, .tar.gz)
copy-pgindent.patch (application/octet-stream, 79.7K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.245
diff -c -r1.245 copy.c
*** src/backend/commands/copy.c 2 Jun 2005 01:21:22 -0000 1.245
--- src/backend/commands/copy.c 27 Jun 2005 18:27:36 -0000
***************
*** 1,3 ****
--- 1,4 ----
+
/*-------------------------------------------------------------------------
*
* copy.c
***************
*** 101,109 ****
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname; /* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname; /* current att for error messages */
/*
--- 102,110 ----
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname; /* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname; /* current att for error messages */
/*
***************
*** 130,154 ****
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line, bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
! List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
! List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char * quote, char * escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
! char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
--- 131,156 ----
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line,
! bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char *quote, char *escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
! char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
***************
*** 181,187 ****
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /* new way */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
--- 183,191 ----
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /*
! * new way
! */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
***************
*** 197,221 ****
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /* old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
! /* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
else
{
! /* very old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
! /* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
--- 201,235 ----
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /*
! * old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
! /*
! * grottiness needed for old COPY OUT protocol
! */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
else
{
! /*
! * very old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
! /*
! * grottiness needed for old COPY OUT protocol
! */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
***************
*** 226,232 ****
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /* new way */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
--- 240,248 ----
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /*
! * new way
! */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
***************
*** 242,266 ****
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /* old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
else
{
! /* very old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
! /* We *must* flush here to ensure FE knows it can send. */
pq_flush();
}
--- 258,290 ----
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /*
! * old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
else
{
! /*
! * very old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
! /*
! * We *must* flush here to ensure FE knows it can send.
! */
pq_flush();
}
***************
*** 271,290 ****
{
if (binary)
{
! /* Need to flush out file trailer word */
CopySendEndOfRow(true);
}
else
{
! /* Shouldn't have any unsent data */
Assert(copy_msgbuf->len == 0);
}
! /* Send Copy Done message */
pq_putemptymessage('c');
}
else
{
! /* The FE/BE protocol uses \n as newline for all platforms */
CopySendData("\\.\n", 3);
pq_endcopyout(false);
}
--- 295,322 ----
{
if (binary)
{
! /*
! * Need to flush out file trailer word
! */
CopySendEndOfRow(true);
}
else
{
! /*
! * Shouldn't have any unsent data
! */
Assert(copy_msgbuf->len == 0);
}
! /*
! * Send Copy Done message
! */
pq_putemptymessage('c');
}
else
{
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
CopySendData("\\.\n", 3);
pq_endcopyout(false);
}
***************
*** 314,323 ****
case COPY_OLD_FE:
if (pq_putbytes((char *) databuf, datasize))
{
! /* no hope of recovering connection sync, so FATAL */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
--- 346,357 ----
case COPY_OLD_FE:
if (pq_putbytes((char *) databuf, datasize))
{
! /*
! * no hope of recovering connection sync, so FATAL
! */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
***************
*** 346,352 ****
case COPY_FILE:
if (!binary)
{
! /* Default line termination depends on platform */
#ifndef WIN32
CopySendChar('\n');
#else
--- 380,388 ----
case COPY_FILE:
if (!binary)
{
! /*
! * Default line termination depends on platform
! */
#ifndef WIN32
CopySendChar('\n');
#else
***************
*** 355,371 ****
}
break;
case COPY_OLD_FE:
! /* The FE/BE protocol uses \n as newline for all platforms */
if (!binary)
CopySendChar('\n');
break;
case COPY_NEW_FE:
! /* The FE/BE protocol uses \n as newline for all platforms */
if (!binary)
CopySendChar('\n');
! /* Dump the accumulated row as one CopyData message */
(void) pq_putmessage('d', copy_msgbuf->data, copy_msgbuf->len);
! /* Reset copy_msgbuf to empty */
copy_msgbuf->len = 0;
copy_msgbuf->data[0] = '\0';
break;
--- 391,415 ----
}
break;
case COPY_OLD_FE:
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
if (!binary)
CopySendChar('\n');
break;
case COPY_NEW_FE:
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
if (!binary)
CopySendChar('\n');
! /*
! * Dump the accumulated row as one CopyData message
! */
(void) pq_putmessage('d', copy_msgbuf->data, copy_msgbuf->len);
! /*
! * Reset copy_msgbuf to empty
! */
copy_msgbuf->len = 0;
copy_msgbuf->data[0] = '\0';
break;
***************
*** 397,403 ****
case COPY_OLD_FE:
if (pq_getbytes((char *) databuf, datasize))
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
--- 441,449 ----
case COPY_OLD_FE:
if (pq_getbytes((char *) databuf, datasize))
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
***************
*** 410,441 ****
while (copy_msgbuf->cursor >= copy_msgbuf->len)
{
! /* Try to receive another message */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
break;
case 'c': /* CopyDone */
! /* COPY IN correctly terminated by frontend */
fe_eof = true;
return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
--- 456,493 ----
while (copy_msgbuf->cursor >= copy_msgbuf->len)
{
! /*
! * Try to receive another message
! */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg
! ("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg
! ("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
break;
case 'c': /* CopyDone */
! /*
! * COPY IN correctly terminated by frontend
! */
fe_eof = true;
return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
***************
*** 450,457 ****
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
! errmsg("unexpected message type 0x%02X during COPY from stdin",
! mtype)));
break;
}
}
--- 502,510 ----
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
! errmsg
! ("unexpected message type 0x%02X during COPY from stdin",
! mtype)));
break;
}
}
***************
*** 480,486 ****
ch = pq_getbyte();
if (ch == EOF)
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
--- 533,541 ----
ch = pq_getbyte();
if (ch == EOF)
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
***************
*** 529,535 ****
ch = pq_peekbyte();
if (ch == EOF)
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
--- 584,592 ----
ch = pq_peekbyte();
if (ch == EOF)
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
***************
*** 565,579 ****
case COPY_FILE:
if (!pickup)
{
! /* We don't want to pick it up - so put it back in there */
ungetc(c, copy_file);
}
! /* If we wanted to pick it up, it's already done */
break;
case COPY_OLD_FE:
if (pickup)
{
! /* We want to pick it up */
(void) pq_getbyte();
}
--- 622,642 ----
case COPY_FILE:
if (!pickup)
{
! /*
! * We don't want to pick it up - so put it back in there
! */
ungetc(c, copy_file);
}
! /*
! * If we wanted to pick it up, it's already done
! */
break;
case COPY_OLD_FE:
if (pickup)
{
! /*
! * We want to pick it up
! */
(void) pq_getbyte();
}
***************
*** 585,594 ****
case COPY_NEW_FE:
if (!pickup)
{
! /* We don't want to pick it up - so put it back in there */
copy_msgbuf->cursor--;
}
! /* If we wanted to pick it up, it's already done */
break;
}
}
--- 648,661 ----
case COPY_NEW_FE:
if (!pickup)
{
! /*
! * We don't want to pick it up - so put it back in there
! */
copy_msgbuf->cursor--;
}
! /*
! * If we wanted to pick it up, it's already done
! */
break;
}
}
***************
*** 695,701 ****
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
--- 762,768 ----
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
***************
*** 708,714 ****
AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
AclResult aclresult;
! /* Extract options from the statement node tree */
foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 775,783 ----
AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
AclResult aclresult;
! /*
! * Extract options from the statement node tree
! */
foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
***************
*** 794,801 ****
force_notnull = (List *) defel->arg;
}
else
! elog(ERROR, "option \"%s\" not recognized",
! defel->defname);
}
if (binary && delim)
--- 863,869 ----
force_notnull = (List *) defel->arg;
}
else
! elog(ERROR, "option \"%s\" not recognized", defel->defname);
}
if (binary && delim)
***************
*** 813,819 ****
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify NULL in BINARY mode")));
! /* Set defaults */
if (!delim)
delim = csv_mode ? "," : "\t";
--- 881,889 ----
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify NULL in BINARY mode")));
! /*
! * Set defaults
! */
if (!delim)
delim = csv_mode ? "," : "\t";
***************
*** 828,846 ****
escape = quote;
}
! /* Only single-character delimiter strings are supported. */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /* Check header */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY HEADER available only in CSV mode")));
! /* Check quote */
if (!csv_mode && quote != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 898,922 ----
escape = quote;
}
! /*
! * Only single-character delimiter strings are supported.
! */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /*
! * Check header
! */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY HEADER available only in CSV mode")));
! /*
! * Check quote
! */
if (!csv_mode && quote != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 851,857 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote must be a single character")));
! /* Check escape */
if (!csv_mode && escape != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 927,935 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote must be a single character")));
! /*
! * Check escape
! */
if (!csv_mode && escape != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 862,868 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape must be a single character")));
! /* Check force_quote */
if (!csv_mode && force_quote != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 940,948 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape must be a single character")));
! /*
! * Check force_quote
! */
if (!csv_mode && force_quote != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 870,909 ****
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
! /* Check force_notnull */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null only available using COPY FROM")));
! /* Don't allow the delimiter to appear in the null string. */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY delimiter must not appear in the NULL specification")));
! /* Don't allow the csv quote char to appear in the null string. */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("CSV quote character must not appear in the NULL specification")));
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(relation, (is_from ? RowExclusiveLock : AccessShareLock));
! /* check read-only transaction */
! if (XactReadOnly && !is_from && !isTempNamespace(RelationGetNamespace(rel)))
ereport(ERROR,
(errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION),
errmsg("transaction is read-only")));
! /* Check permissions. */
aclresult = pg_class_aclcheck(RelationGetRelid(rel), GetUserId(),
required_access);
if (aclresult != ACLCHECK_OK)
--- 950,1006 ----
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
! /*
! * Check force_notnull
! */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY force not null only available using COPY FROM")));
! /*
! * Don't allow the delimiter to appear in the null string.
! */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY delimiter must not appear in the NULL specification")));
! /*
! * Don't allow the csv quote char to appear in the null string.
! */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("CSV quote character must not appear in the NULL specification")));
! /*
! * Open and lock the relation, using the appropriate lock type.
! */
! rel =
! heap_openrv(relation, (is_from ? RowExclusiveLock : AccessShareLock));
! /*
! * check read-only transaction
! */
! if (XactReadOnly && !is_from
! && !isTempNamespace(RelationGetNamespace(rel)))
ereport(ERROR,
(errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION),
errmsg("transaction is read-only")));
! /*
! * Check permissions.
! */
aclresult = pg_class_aclcheck(RelationGetRelid(rel), GetUserId(),
required_access);
if (aclresult != ACLCHECK_OK)
***************
*** 914,932 ****
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
! /* Don't allow COPY w/ OIDs to or from a table without them */
if (oids && !rel->rd_rel->relhasoids)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("table \"%s\" does not have OIDs",
RelationGetRelationName(rel))));
! /* Generate or convert list of attributes to process */
attnumlist = CopyGetAttnums(rel, attnamelist);
! /* Check that FORCE QUOTE references valid COPY columns */
if (force_quote)
{
TupleDesc tupDesc = RelationGetDescr(rel);
--- 1011,1035 ----
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
! /*
! * Don't allow COPY w/ OIDs to or from a table without them
! */
if (oids && !rel->rd_rel->relhasoids)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("table \"%s\" does not have OIDs",
RelationGetRelationName(rel))));
! /*
! * Generate or convert list of attributes to process
! */
attnumlist = CopyGetAttnums(rel, attnamelist);
! /*
! * Check that FORCE QUOTE references valid COPY columns
! */
if (force_quote)
{
TupleDesc tupDesc = RelationGetDescr(rel);
***************
*** 942,953 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /* Check that FORCE NOT NULL references valid COPY columns */
if (force_notnull)
{
ListCell *cur;
--- 1045,1059 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg
! ("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /*
! * Check that FORCE NOT NULL references valid COPY columns
! */
if (force_notnull)
{
ListCell *cur;
***************
*** 963,974 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /* Set up variables to avoid per-attribute overhead. */
initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
--- 1069,1083 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg
! ("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /*
! * Set up variables to avoid per-attribute overhead.
! */
initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
***************
*** 998,1005 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1107,1114 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1017,1024 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1126,1133 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1049,1056 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1158,1165 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1071,1077 ****
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
--- 1180,1187 ----
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg
! ("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
***************
*** 1080,1087 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1190,1197 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1099,1110 ****
if (!pipe)
{
! /* we assume only the write case could fail here */
if (FreeFile(copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
! filename)));
}
pfree(attribute_buf.data);
pfree(line_buf.data);
--- 1209,1221 ----
if (!pipe)
{
! /*
! * we assume only the write case could fail here
! */
if (FreeFile(copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m", filename)));
}
pfree(attribute_buf.data);
pfree(line_buf.data);
***************
*** 1178,1184 ****
num_phys_attrs = tupDesc->natts;
attr_count = list_length(attnumlist);
! /* Get info about the columns we need to process. */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
foreach(cur, attnumlist)
--- 1289,1297 ----
num_phys_attrs = tupDesc->natts;
attr_count = list_length(attnumlist);
! /*
! * Get info about the columns we need to process.
! */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
foreach(cur, attnumlist)
***************
*** 1189,1200 ****
if (binary)
getTypeBinaryOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid,
! &isvarlena);
else
getTypeOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid,
! &isvarlena);
fmgr_info(out_func_oid, &out_functions[attnum - 1]);
if (list_member_int(force_quote_atts, attnum))
--- 1302,1311 ----
if (binary)
getTypeBinaryOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid, &isvarlena);
else
getTypeOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid, &isvarlena);
fmgr_info(out_func_oid, &out_functions[attnum - 1]);
if (list_member_int(force_quote_atts, attnum))
***************
*** 1217,1233 ****
if (binary)
{
! /* Generate header for a binary copy */
int32 tmp;
! /* Signature */
CopySendData((char *) BinarySignature, 11);
! /* Flags field */
tmp = 0;
if (oids)
tmp |= (1 << 16);
CopySendInt32(tmp);
! /* No header extension */
tmp = 0;
CopySendInt32(tmp);
}
--- 1328,1352 ----
if (binary)
{
! /*
! * Generate header for a binary copy
! */
int32 tmp;
! /*
! * Signature
! */
CopySendData((char *) BinarySignature, 11);
! /*
! * Flags field
! */
tmp = 0;
if (oids)
tmp |= (1 << 16);
CopySendInt32(tmp);
! /*
! * No header extension
! */
tmp = 0;
CopySendInt32(tmp);
}
***************
*** 1242,1253 ****
pg_server_to_client((unsigned char *) null_print,
strlen(null_print));
! /* if a header has been requested send the line */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 1361,1374 ----
pg_server_to_client((unsigned char *) null_print,
strlen(null_print));
! /*
! * if a header has been requested send the line
! */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1280,1304 ****
if (binary)
{
! /* Binary per-tuple header */
CopySendInt16(attr_count);
! /* Send OID if wanted --- note attr_count doesn't include it */
if (oids)
{
Oid oid = HeapTupleGetOid(tuple);
! /* Hack --- assume Oid is same size as int32 */
CopySendInt32(sizeof(int32));
CopySendInt32(oid);
}
}
else
{
! /* Text format has no per-tuple header, but send OID if wanted */
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum(HeapTupleGetOid(tuple))));
CopySendString(string);
need_delim = true;
}
--- 1401,1435 ----
if (binary)
{
! /*
! * Binary per-tuple header
! */
CopySendInt16(attr_count);
! /*
! * Send OID if wanted --- note attr_count doesn't include it
! */
if (oids)
{
Oid oid = HeapTupleGetOid(tuple);
! /*
! * Hack --- assume Oid is same size as int32
! */
CopySendInt32(sizeof(int32));
CopySendInt32(oid);
}
}
else
{
! /*
! * Text format has no per-tuple header, but send OID if wanted
! */
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum
! (HeapTupleGetOid
! (tuple))));
CopySendString(string);
need_delim = true;
}
***************
*** 1330,1342 ****
{
if (!binary)
{
! string = DatumGetCString(FunctionCall1(&out_functions[attnum - 1],
! value));
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
--- 1461,1474 ----
{
if (!binary)
{
! string =
! DatumGetCString(FunctionCall1
! (&out_functions[attnum - 1], value));
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
***************
*** 1346,1354 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
! /* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
VARSIZE(outputbytes) - VARHDRSZ);
--- 1478,1489 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1
! (&out_functions[attnum - 1], value));
! /*
! * We assume the result will not have been toasted
! */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
VARSIZE(outputbytes) - VARHDRSZ);
***************
*** 1365,1371 ****
if (binary)
{
! /* Generate trailer for a binary copy */
CopySendInt16(-1);
}
--- 1500,1508 ----
if (binary)
{
! /*
! * Generate trailer for a binary copy
! */
CopySendInt16(-1);
}
***************
*** 1384,1390 ****
{
if (copy_binary)
{
! /* can't usefully display the data */
if (copy_attname)
errcontext("COPY %s, line %d, column %s",
copy_relname, copy_lineno, copy_attname);
--- 1521,1529 ----
{
if (copy_binary)
{
! /*
! * can't usefully display the data
! */
if (copy_attname)
errcontext("COPY %s, line %d, column %s",
copy_relname, copy_lineno, copy_attname);
***************
*** 1395,1401 ****
{
if (copy_attname)
{
! /* error is relevant to a particular column */
limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
--- 1534,1542 ----
{
if (copy_attname)
{
! /*
! * error is relevant to a particular column
! */
limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
***************
*** 1403,1423 ****
}
else
{
! /* error is relevant to a particular line */
! if (line_buf_converted ||
! client_encoding == server_encoding)
{
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
! copy_relname, copy_lineno,
! line_buf.data);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
--- 1544,1564 ----
}
else
{
! /*
! * error is relevant to a particular line
! */
! if (line_buf_converted || client_encoding == server_encoding)
{
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
! copy_relname, copy_lineno, line_buf.data);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
***************
*** 1445,1462 ****
int len;
! /* Fast path if definitely okay */
if (buf->len <= MAX_COPY_DATA_DISPLAY)
return;
! /* Apply encoding-dependent truncation */
len = pg_mbcliplen(buf->data, buf->len, MAX_COPY_DATA_DISPLAY);
if (buf->len <= len)
return; /* no need to truncate */
buf->len = len;
buf->data[len] = '\0';
! /* Add "..." to show we truncated the input */
appendStringInfoString(buf, "...");
}
--- 1586,1609 ----
int len;
! /*
! * Fast path if definitely okay
! */
if (buf->len <= MAX_COPY_DATA_DISPLAY)
return;
! /*
! * Apply encoding-dependent truncation
! */
len = pg_mbcliplen(buf->data, buf->len, MAX_COPY_DATA_DISPLAY);
if (buf->len <= len)
return; /* no need to truncate */
buf->len = len;
buf->data[len] = '\0';
! /*
! * Add "..." to show we truncated the input
! */
appendStringInfoString(buf, "...");
}
***************
*** 1510,1521 ****
* code here that basically duplicated execUtils.c ...)
*/
resultRelInfo = makeNode(ResultRelInfo);
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
resultRelInfo->ri_RelationDesc = rel;
resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers * sizeof(FmgrInfo));
resultRelInfo->ri_TrigInstrument = NULL;
ExecOpenIndices(resultRelInfo);
--- 1657,1670 ----
* code here that basically duplicated execUtils.c ...)
*/
resultRelInfo = makeNode(ResultRelInfo);
+
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
resultRelInfo->ri_RelationDesc = rel;
resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers *
! sizeof(FmgrInfo));
resultRelInfo->ri_TrigInstrument = NULL;
ExecOpenIndices(resultRelInfo);
***************
*** 1524,1530 ****
estate->es_num_result_relations = 1;
estate->es_result_relation_info = resultRelInfo;
! /* Set up a tuple slot too */
slot = MakeSingleTupleTableSlot(tupDesc);
econtext = GetPerTupleExprContext(estate);
--- 1673,1681 ----
estate->es_num_result_relations = 1;
estate->es_result_relation_info = resultRelInfo;
! /*
! * Set up a tuple slot too
! */
slot = MakeSingleTupleTableSlot(tupDesc);
econtext = GetPerTupleExprContext(estate);
***************
*** 1539,1557 ****
typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
defmap = (int *) palloc(num_phys_attrs * sizeof(int));
defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs = (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
for (attnum = 1; attnum <= num_phys_attrs; attnum++)
{
! /* We don't need info for dropped attributes */
if (attr[attnum - 1]->attisdropped)
continue;
! /* Fetch the input function and typioparam info */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
--- 1690,1713 ----
typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
defmap = (int *) palloc(num_phys_attrs * sizeof(int));
defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs =
! (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
for (attnum = 1; attnum <= num_phys_attrs; attnum++)
{
! /*
! * We don't need info for dropped attributes
! */
if (attr[attnum - 1]->attisdropped)
continue;
! /*
! * Fetch the input function and typioparam info
! */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
***************
*** 1562,1572 ****
else
force_notnull[attnum - 1] = false;
! /* Get default info if needed */
if (!list_member_int(attnumlist, attnum))
{
! /* attribute is NOT to be copied from input */
! /* use default value if one exists */
Node *defexpr = build_column_default(rel, attnum);
if (defexpr != NULL)
--- 1718,1734 ----
else
force_notnull[attnum - 1] = false;
! /*
! * Get default info if needed
! */
if (!list_member_int(attnumlist, attnum))
{
! /*
! * attribute is NOT to be copied from input
! */
! /*
! * use default value if one exists
! */
Node *defexpr = build_column_default(rel, attnum);
if (defexpr != NULL)
***************
*** 1578,1584 ****
}
}
! /* If it's a domain type, set up to check domain constraints */
if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
{
Param *prm;
--- 1740,1748 ----
}
}
! /*
! * If it's a domain type, set up to check domain constraints
! */
if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
{
Param *prm;
***************
*** 1593,1598 ****
--- 1757,1763 ----
* actual datum during the data input loop.
*/
prm = makeNode(Param);
+
prm->paramkind = PARAM_EXEC;
prm->paramid = 0;
prm->paramtype = getBaseType(attr[attnum - 1]->atttypid);
***************
*** 1608,1614 ****
}
}
! /* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
/*
--- 1773,1781 ----
}
}
! /*
! * Prepare to catch AFTER triggers.
! */
AfterTriggerBeginQuery();
/*
***************
*** 1624,1640 ****
file_has_oids = oids; /* must rely on user to tell us this... */
else
{
! /* Read and verify binary header */
char readSig[11];
int32 tmp;
! /* Signature */
CopyGetData(readSig, 11);
if (CopyGetEof() || memcmp(readSig, BinarySignature, 11) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("COPY file signature not recognized")));
! /* Flags field */
tmp = CopyGetInt32();
if (CopyGetEof())
ereport(ERROR,
--- 1791,1813 ----
file_has_oids = oids; /* must rely on user to tell us this... */
else
{
! /*
! * Read and verify binary header
! */
char readSig[11];
int32 tmp;
! /*
! * Signature
! */
CopyGetData(readSig, 11);
if (CopyGetEof() || memcmp(readSig, BinarySignature, 11) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("COPY file signature not recognized")));
! /*
! * Flags field
! */
tmp = CopyGetInt32();
if (CopyGetEof())
ereport(ERROR,
***************
*** 1645,1684 ****
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
while (tmp-- > 0)
{
CopyGetData(readSig, 1);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
if (file_has_oids && binary)
{
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
fmgr_info(in_func_oid, &oid_in_function);
}
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! /* Make room for a PARAM_EXEC value for domain constraint checks */
if (hasConstraints)
econtext->ecxt_param_exec_vals = (ParamExecData *)
palloc0(sizeof(ParamExecData));
! /* Initialize static variables */
fe_eof = false;
eol_type = EOL_UNKNOWN;
copy_binary = binary;
--- 1818,1865 ----
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unrecognized critical flags in COPY file header")));
! /*
! * Header extension length
! */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /*
! * Skip extension header, if present
! */
while (tmp-- > 0)
{
CopyGetData(readSig, 1);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
if (file_has_oids && binary)
{
! getTypeBinaryInputInfo(OIDOID, &in_func_oid, &oid_typioparam);
fmgr_info(in_func_oid, &oid_in_function);
}
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! /*
! * Make room for a PARAM_EXEC value for domain constraint checks
! */
if (hasConstraints)
econtext->ecxt_param_exec_vals = (ParamExecData *)
palloc0(sizeof(ParamExecData));
! /*
! * Initialize static variables
! */
fe_eof = false;
eol_type = EOL_UNKNOWN;
copy_binary = binary;
***************
*** 1686,1702 ****
copy_lineno = 0;
copy_attname = NULL;
! /* Set up callback to identify error line number */
errcontext.callback = copy_in_error_callback;
errcontext.arg = NULL;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /* on input just throw the header line away */
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape) ;
}
while (!done)
--- 1867,1887 ----
copy_lineno = 0;
copy_attname = NULL;
! /*
! * Set up callback to identify error line number
! */
errcontext.callback = copy_in_error_callback;
errcontext.arg = NULL;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /*
! * on input just throw the header line away
! */
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape);
}
while (!done)
***************
*** 1708,1720 ****
copy_lineno++;
! /* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
! /* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
--- 1893,1911 ----
copy_lineno++;
! /*
! * Reset the per-tuple exprcontext
! */
ResetPerTupleExprContext(estate);
! /*
! * Switch into its memory context
! */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /*
! * Initialize all values for row to NULL
! */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
***************
*** 1724,1731 ****
char *string;
ListCell *cur;
! /* Actually read the line into memory here */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
--- 1915,1924 ----
char *string;
ListCell *cur;
! /*
! * Actually read the line into memory here
! */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
***************
*** 1739,1745 ****
if (file_has_oids)
{
! /* can't be in CSV mode here */
string = CopyReadAttribute(delim, null_print,
&result, &isnull);
--- 1932,1940 ----
if (file_has_oids)
{
! /*
! * can't be in CSV mode here
! */
string = CopyReadAttribute(delim, null_print,
&result, &isnull);
***************
*** 1751,1757 ****
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 1946,1953 ----
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum
! (string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 1760,1766 ****
}
}
! /* Loop to read the user attributes on the line. */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 1956,1964 ----
}
}
! /*
! * Loop to read the user attributes on the line.
! */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1779,1789 ****
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
--- 1977,1987 ----
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
***************
*** 1795,1808 ****
isnull = false;
}
! /* we read an SQL NULL, no need to do anything */
if (!isnull)
{
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
--- 1993,2009 ----
isnull = false;
}
! /*
! * we read an SQL NULL, no need to do anything
! */
if (!isnull)
{
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->
! atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
***************
*** 1818,1828 ****
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
! /* binary */
int16 fld_count;
ListCell *cur;
--- 2019,2031 ----
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
! /*
! * binary
! */
int16 fld_count;
ListCell *cur;
***************
*** 1844,1851 ****
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
--- 2047,2054 ----
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
***************
*** 1864,1871 ****
i++;
values[m] = CopyReadBinaryAttribute(i,
&in_functions[m],
! typioparams[m],
! &isnull);
nulls[m] = isnull ? 'n' : ' ';
copy_attname = NULL;
}
--- 2067,2073 ----
i++;
values[m] = CopyReadBinaryAttribute(i,
&in_functions[m],
! typioparams[m], &isnull);
nulls[m] = isnull ? 'n' : ' ';
copy_attname = NULL;
}
***************
*** 1884,1890 ****
nulls[defmap[i]] = ' ';
}
! /* Next apply any domain constraints */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
--- 2086,2094 ----
nulls[defmap[i]] = ' ';
}
! /*
! * Next apply any domain constraints
! */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
***************
*** 1896,1902 ****
if (exprstate == NULL)
continue; /* no constraint for this attr */
! /* Insert current row's value into the Param value */
prmdata->value = values[i];
prmdata->isnull = (nulls[i] == 'n');
--- 2100,2108 ----
if (exprstate == NULL)
continue; /* no constraint for this attr */
! /*
! * Insert current row's value into the Param value
! */
prmdata->value = values[i];
prmdata->isnull = (nulls[i] == 'n');
***************
*** 1905,1928 ****
* expression to replace the value (consider e.g. a
* timestamp precision restriction).
*/
! values[i] = ExecEvalExpr(exprstate, econtext,
! &isnull, NULL);
nulls[i] = isnull ? 'n' : ' ';
}
}
! /* And now we can form the input tuple. */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /* Triggers and stuff need to be invoked in query context. */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
! /* BEFORE ROW INSERT Triggers */
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
--- 2111,2139 ----
* expression to replace the value (consider e.g. a
* timestamp precision restriction).
*/
! values[i] = ExecEvalExpr(exprstate, econtext, &isnull, NULL);
nulls[i] = isnull ? 'n' : ' ';
}
}
! /*
! * And now we can form the input tuple.
! */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /*
! * Triggers and stuff need to be invoked in query context.
! */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
! /*
! * BEFORE ROW INSERT Triggers
! */
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
***************
*** 1941,1973 ****
if (!skip_tuple)
{
! /* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /* Check the constraints of the tuple */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /* OK, store the tuple and create index entries for it */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
! /* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /* Done, clean up */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /* Execute AFTER STATEMENT insertion triggers */
ExecASInsertTriggers(estate, resultRelInfo);
! /* Handle queued AFTER triggers */
AfterTriggerEndQuery(estate);
pfree(values);
--- 2152,2198 ----
if (!skip_tuple)
{
! /*
! * Place tuple in tuple slot
! */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /*
! * Check the constraints of the tuple
! */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /*
! * OK, store the tuple and create index entries for it
! */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
! /*
! * AFTER ROW INSERT Triggers
! */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /*
! * Done, clean up
! */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
ExecASInsertTriggers(estate, resultRelInfo);
! /*
! * Handle queued AFTER triggers
! */
AfterTriggerEndQuery(estate);
pfree(values);
***************
*** 1996,2002 ****
* by newline.
*/
static bool
! CopyReadLine(char * quote, char * escape)
{
bool result;
bool change_encoding = (client_encoding != server_encoding);
--- 2221,2227 ----
* by newline.
*/
static bool
! CopyReadLine(char *quote, char *escape)
{
bool result;
bool change_encoding = (client_encoding != server_encoding);
***************
*** 2005,2019 ****
int j;
unsigned char s[2];
char *cvt;
! bool in_quote = false, last_was_esc = false, csv_mode = false;
! char quotec = '\0', escapec = '\0';
if (quote)
{
csv_mode = true;
quotec = quote[0];
escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
}
--- 2230,2249 ----
int j;
unsigned char s[2];
char *cvt;
! bool in_quote = false,
! last_was_esc = false,
! csv_mode = false;
! char quotec = '\0',
! escapec = '\0';
if (quote)
{
csv_mode = true;
quotec = quote[0];
escapec = escape[0];
! /*
! * ignore special escape processing if it's the same as quotec
! */
if (quotec == escapec)
escapec = '\0';
}
***************
*** 2021,2051 ****
s[1] = 0;
! /* reset line_buf to empty */
line_buf.len = 0;
line_buf.data[0] = '\0';
line_buf.cursor = 0;
! /* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /* set default status */
result = false;
/*
* In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
*
* In Text mode, for backwards compatibility we allow
* backslashes to escape newline characters. Backslashes other than
* the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
*
* In CSV mode, CR and NL inside q quoted field are just part of the
* data value and are put in line_buf. We keep just enough state
* to know if we are currently in a quoted field or not.
*
! * These four characters, and only these four, are assumed the same in
* frontend and backend encodings.
*
* We do not assume that second and later bytes of a frontend
--- 2251,2287 ----
s[1] = 0;
! /*
! * reset line_buf to empty
! */
line_buf.len = 0;
line_buf.data[0] = '\0';
line_buf.cursor = 0;
! /*
! * mark that encoding conversion hasn't occurred yet
! */
line_buf_converted = false;
! /*
! * set default status
! */
result = false;
/*
* In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
*
* In Text mode, for backwards compatibility we allow
* backslashes to escape newline characters. Backslashes other than
* the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
*
* In CSV mode, CR and NL inside q quoted field are just part of the
* data value and are put in line_buf. We keep just enough state
* to know if we are currently in a quoted field or not.
*
! * These four characters, and only these four, are assumed the same in
* frontend and backend encodings.
*
* We do not assume that second and later bytes of a frontend
***************
*** 2062,2108 ****
if (csv_mode)
{
! /*
* Dealing with quotes and escapes here is mildly tricky. If the
* quote char is also the escape char, there's no problem - we
* just use the char as a toggle. If they are different, we need
* to ensure that we only take account of an escape inside a quoted
* field and immediately preceding a quote char, and not the
* second in a escape-escape sequence.
! */
if (in_quote && c == escapec)
! last_was_esc = ! last_was_esc;
! if (c == quotec && ! last_was_esc)
! in_quote = ! in_quote;
if (c != escapec)
last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
* the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
}
if (!in_quote && c == '\r')
{
if (eol_type == EOL_NL)
{
! if (! csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
! /* Check for \r\n on first line, _and_ handle \r\n. */
if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
int c2 = CopyPeekChar();
--- 2298,2349 ----
if (csv_mode)
{
! /*
* Dealing with quotes and escapes here is mildly tricky. If the
* quote char is also the escape char, there's no problem - we
* just use the char as a toggle. If they are different, we need
* to ensure that we only take account of an escape inside a quoted
* field and immediately preceding a quote char, and not the
* second in a escape-escape sequence.
! */
if (in_quote && c == escapec)
! last_was_esc = !last_was_esc;
! if (c == quotec && !last_was_esc)
! in_quote = !in_quote;
if (c != escapec)
last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
* the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
}
if (!in_quote && c == '\r')
{
if (eol_type == EOL_NL)
{
! if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("literal carriage return found in data"),
! errhint
! ("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unquoted carriage return found in CSV data"),
! errhint
! ("Use quoted CSV field to represent carriage return.")));
}
! /*
! * Check for \r\n on first line, _and_ handle \r\n.
! */
if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
int c2 = CopyPeekChar();
***************
*** 2114,2132 ****
}
else
{
! /* found \r, but no \n */
if (eol_type == EOL_CRNL)
{
if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
--- 2355,2379 ----
}
else
{
! /*
! * found \r, but no \n
! */
if (eol_type == EOL_CRNL)
{
if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("literal carriage return found in data"),
! errhint
! ("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unquoted carriage return found in data"),
! errhint
! ("Use quoted CSV field to represent carriage return.")));
}
***************
*** 2153,2160 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
}
eol_type = EOL_NL;
break;
--- 2400,2408 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unquoted newline found in data"),
! errhint
! ("Use quoted CSV field to represent newline.")));
!
}
eol_type = EOL_NL;
break;
***************
*** 2162,2169 ****
if ((line_buf.len == 0 || !csv_mode) && c == '\\')
{
! int c2;
!
if (csv_mode)
c2 = CopyPeekChar();
else
--- 2410,2417 ----
if ((line_buf.len == 0 || !csv_mode) && c == '\\')
{
! int c2;
!
if (csv_mode)
c2 = CopyPeekChar();
else
***************
*** 2179,2185 ****
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
if (eol_type == EOL_CRNL)
{
--- 2427,2433 ----
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
if (eol_type == EOL_CRNL)
{
***************
*** 2187,2193 ****
if (c == '\n')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
if (c != '\r')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 2435,2442 ----
if (c == '\n')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("end-of-copy marker does not match previous newline style")));
if (c != '\r')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 2203,2209 ****
(eol_type == EOL_CR && c != '\r'))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
/*
* In protocol version 3, we should ignore anything after
--- 2452,2459 ----
(eol_type == EOL_CR && c != '\r'))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("end-of-copy marker does not match previous newline style")));
/*
* In protocol version 3, we should ignore anything after
***************
*** 2218,2228 ****
result = true; /* report EOF */
break;
}
!
if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /* not EOF mark, so emit \ and following char literally */
appendStringInfoCharMacro(&line_buf, '\\');
}
--- 2468,2480 ----
result = true; /* report EOF */
break;
}
!
if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /*
! * not EOF mark, so emit \ and following char literally
! */
appendStringInfoCharMacro(&line_buf, '\\');
}
***************
*** 2254,2274 ****
}
} /* end of outer loop */
! /* Done reading the line. Convert it to server encoding. */
if (change_encoding)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
if (cvt != line_buf.data)
{
! /* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* Now it's safe to use the buffer in error messages */
line_buf_converted = true;
return result;
--- 2506,2532 ----
}
} /* end of outer loop */
! /*
! * Done reading the line. Convert it to server encoding.
! */
if (change_encoding)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
if (cvt != line_buf.data)
{
! /*
! * transfer converted data back to line_buf
! */
line_buf.len = 0;
line_buf.data[0] = '\0';
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /*
! * Now it's safe to use the buffer in error messages
! */
line_buf_converted = true;
return result;
***************
*** 2277,2284 ****
/*
* Return decimal value for a hexadecimal digit
*/
! static
! int GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
--- 2535,2542 ----
/*
* Return decimal value for a hexadecimal digit
*/
! static int
! GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
***************
*** 2314,2324 ****
int end_cursor;
int input_len;
! /* reset attribute_buf to empty */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
! /* set default status */
*result = END_OF_LINE;
for (;;)
--- 2572,2586 ----
int end_cursor;
int input_len;
! /*
! * reset attribute_buf to empty
! */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
! /*
! * set default status
! */
*result = END_OF_LINE;
for (;;)
***************
*** 2347,2353 ****
case '5':
case '6':
case '7':
! /* handle \013 */
{
int val;
--- 2609,2617 ----
case '5':
case '6':
case '7':
! /*
! * handle \013
! */
{
int val;
***************
*** 2374,2387 ****
}
break;
case 'x':
! /* Handle \x3F */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
--- 2638,2653 ----
}
break;
case 'x':
! /*
! * Handle \x3F
! */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
***************
*** 2390,2396 ****
if (isxdigit(hexchar))
{
line_buf.cursor++;
! val = (val << 4) + GetDecimalFromHex(hexchar);
}
}
c = val & 0xff;
--- 2656,2664 ----
if (isxdigit(hexchar))
{
line_buf.cursor++;
! val =
! (val << 4) +
! GetDecimalFromHex(hexchar);
}
}
c = val & 0xff;
***************
*** 2425,2431 ****
appendStringInfoCharMacro(&attribute_buf, c);
}
! /* check whether raw input matched null marker */
input_len = end_cursor - start_cursor;
if (input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
--- 2693,2701 ----
appendStringInfoCharMacro(&attribute_buf, c);
}
! /*
! * check whether raw input matched null marker
! */
input_len = end_cursor - start_cursor;
if (input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
***************
*** 2477,2487 ****
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
! /* set default status */
*result = END_OF_LINE;
for (;;)
--- 2747,2761 ----
bool in_quote = false;
bool saw_quote = false;
! /*
! * reset attribute_buf to empty
! */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
! /*
! * set default status
! */
*result = END_OF_LINE;
for (;;)
***************
*** 2491,2504 ****
break;
c = line_buf.data[line_buf.cursor++];
! /* unquoted field delimiter */
if (!in_quote && c == delimc)
{
*result = NORMAL_ATTR;
break;
}
! /* start of quoted field (or part of field) */
if (!in_quote && c == quotec)
{
saw_quote = true;
--- 2765,2782 ----
break;
c = line_buf.data[line_buf.cursor++];
! /*
! * unquoted field delimiter
! */
if (!in_quote && c == delimc)
{
*result = NORMAL_ATTR;
break;
}
! /*
! * start of quoted field (or part of field)
! */
if (!in_quote && c == quotec)
{
saw_quote = true;
***************
*** 2506,2512 ****
continue;
}
! /* escape within a quoted field */
if (in_quote && c == escapec)
{
/*
--- 2784,2792 ----
continue;
}
! /*
! * escape within a quoted field
! */
if (in_quote && c == escapec)
{
/*
***************
*** 2542,2548 ****
if (in_quote)
*result = UNTERMINATED_FIELD;
! /* check whether raw input matched null marker */
input_len = end_cursor - start_cursor;
if (!saw_quote && input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
--- 2822,2830 ----
if (in_quote)
*result = UNTERMINATED_FIELD;
! /*
! * check whether raw input matched null marker
! */
input_len = end_cursor - start_cursor;
if (!saw_quote && input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
***************
*** 2578,2584 ****
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
attribute_buf.cursor = 0;
--- 2860,2868 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /*
! * reset attribute_buf to empty, and load raw data in it
! */
attribute_buf.len = 0;
attribute_buf.data[0] = '\0';
attribute_buf.cursor = 0;
***************
*** 2594,2605 ****
attribute_buf.len = fld_size;
attribute_buf.data[fld_size] = '\0';
! /* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
PointerGetDatum(&attribute_buf),
ObjectIdGetDatum(typioparam));
! /* Trouble if it didn't eat the whole buffer */
if (attribute_buf.cursor != attribute_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
--- 2878,2893 ----
attribute_buf.len = fld_size;
attribute_buf.data[fld_size] = '\0';
! /*
! * Call the column type's binary input converter
! */
result = FunctionCall2(flinfo,
PointerGetDatum(&attribute_buf),
ObjectIdGetDatum(typioparam));
! /*
! * Trouble if it didn't eat the whole buffer
! */
if (attribute_buf.cursor != attribute_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
***************
*** 2668,2674 ****
*/
if (!same_encoding)
{
! /* send additional bytes of the char, if any */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
--- 2956,2964 ----
*/
if (!same_encoding)
{
! /*
! * send additional bytes of the char, if any
! */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
***************
*** 2709,2716 ****
*/
for (test_string = string;
! !use_quote && (c = *test_string) != '\0';
! test_string += mblen)
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
use_quote = true;
--- 2999,3005 ----
*/
for (test_string = string;
! !use_quote && (c = *test_string) != '\0'; test_string += mblen)
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
use_quote = true;
***************
*** 2732,2738 ****
if (!same_encoding)
{
! /* send additional bytes of the char, if any */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
--- 3021,3029 ----
if (!same_encoding)
{
! /*
! * send additional bytes of the char, if any
! */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
***************
*** 2759,2765 ****
if (attnamelist == NIL)
{
! /* Generate default column list */
TupleDesc tupDesc = RelationGetDescr(rel);
Form_pg_attribute *attr = tupDesc->attrs;
int attr_count = tupDesc->natts;
--- 3050,3058 ----
if (attnamelist == NIL)
{
! /*
! * Generate default column list
! */
TupleDesc tupDesc = RelationGetDescr(rel);
Form_pg_attribute *attr = tupDesc->attrs;
int attr_count = tupDesc->natts;
***************
*** 2774,2780 ****
}
else
{
! /* Validate the user-supplied list and extract attnums */
ListCell *l;
foreach(l, attnamelist)
--- 3067,3075 ----
}
else
{
! /*
! * Validate the user-supplied list and extract attnums
! */
ListCell *l;
foreach(l, attnamelist)
***************
*** 2782,2791 ****
char *name = strVal(lfirst(l));
int attnum;
! /* Lookup column name, ereport on failure */
! /* Note we disallow system columns here */
attnum = attnameAttNum(rel, name, false);
! /* Check for duplicates */
if (list_member_int(attnums, attnum))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_COLUMN),
--- 3077,3092 ----
char *name = strVal(lfirst(l));
int attnum;
! /*
! * Lookup column name, ereport on failure
! */
! /*
! * Note we disallow system columns here
! */
attnum = attnameAttNum(rel, name, false);
! /*
! * Check for duplicates
! */
if (list_member_int(attnums, attnum))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_COLUMN),
fast_copy_V12_8.1.patch (application/octet-stream, 125.9K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.245
diff -c -r1.245 copy.c
*** src/backend/commands/copy.c 2 Jun 2005 01:21:22 -0000 1.245
--- src/backend/commands/copy.c 27 Jun 2005 18:45:39 -0000
***************
*** 1,3 ****
--- 1,4 ----
+
/*-------------------------------------------------------------------------
*
* copy.c
***************
*** 51,56 ****
--- 52,58 ----
#define ISOCTAL(c) (((c) >= '0') && ((c) <= '7'))
#define OCTVALUE(c) ((c) - '0')
+ #define COPY_BUF_SIZE 65536
/*
* Represents the different source/dest cases we need to worry about at
***************
*** 84,90 ****
EOL_CRNL
} EolType;
-
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
/*
--- 86,91 ----
***************
*** 92,97 ****
--- 93,101 ----
* never been reentrant...
*/
static CopyDest copy_dest;
+ static int eol_ch[2]; /* The byte values of the 1 or 2 eol bytes */
+ static char escapec; /* escape char for delimited data format. */
+ static bool client_encoding_only; /* true if client encoding is a non supported server encoding */
static FILE *copy_file; /* used if copy_dest == COPY_FILE */
static StringInfo copy_msgbuf; /* used if copy_dest == COPY_NEW_FE */
static bool fe_eof; /* true if detected end of copy data */
***************
*** 101,156 ****
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname; /* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname; /* current att for error messages */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attribute_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attribute_buf's data buffer!
*/
! static StringInfoData attribute_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attribute_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf;
static bool line_buf_converted;
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line, bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
! List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
! List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char * quote, char * escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
! char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
--- 105,189 ----
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
! static const char *copy_relname; /* table name for error messages */
static int copy_lineno; /* line number for error messages */
! static const char *copy_attname; /* current att for error messages */
!
! /* Static variables for buffered input parsing */
!
! /* leave room for extra null (to enable use of string functions) */
! static char input_buf[COPY_BUF_SIZE + 1];
! static char *begloc;
! static char *endloc;
! static bool esc_in_prevbuf; /* backslash was last character of the data input buffer */
! static bool cr_in_prevbuf; /* CR was last character of the data input buffer */
! static bool line_done; /* finished processing the whole line or stopped in the middle */
! static bool buf_done; /* finished processing the current buffer */
! static int buffer_index; /* input buffer index */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attr_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attr_buf's data buffer!
*/
! static StringInfoData attr_buf; /* still used in CopyFrom() */
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attr_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf; /* still used in CopyFromBinary/CSV() */
static bool line_buf_converted;
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line,
! bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char *quote, char *escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
! char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
+ static bool CopyReadLineBuffered(size_t bytesread);
+ static void CopyFromDelimited(Relation rel, List *attnumlist, bool binary,
+ bool oids, char *delim, char *null_print,
+ bool csv_mode, char *quote, char *escape,
+ List *force_notnull_atts);
+ static char *CopyReadAttribute(const char *delim, const char *null_print,
+ CopyReadResult *result, bool *isnull);
+ static void CopyReadAllAttrs(const char *delim, const char *null_print,
+ int null_print_len, char *nulls, List *attnumlist,
+ int *attr_offsets, int num_phys_attrs,
+ Form_pg_attribute *attr);
+ static char *CopyReadOidAttr(const char *delim, const char *null_print,
+ int null_print_len, CopyReadResult *result,
+ bool *isnull);
+ static bool DetectLineEnd(size_t bytesread);
+
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
***************
*** 160,166 ****
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static void CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
--- 193,199 ----
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static int CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
***************
*** 171,176 ****
--- 204,213 ----
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
+ /* new parsing utils */
+ static char *strchrlen(const char *s, int c, size_t len);
+ static char *str2chrlen(const char *s, int c1, int c2, size_t len,
+ char *c_found);
/*
* Send copy start/stop messages for frontend copies. These have changed
***************
*** 181,187 ****
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /* new way */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
--- 218,226 ----
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /*
! * new way
! */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
***************
*** 197,221 ****
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /* old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
! /* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
else
{
! /* very old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
! /* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
--- 236,270 ----
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /*
! * old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
! /*
! * grottiness needed for old COPY OUT protocol
! */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
else
{
! /*
! * very old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
! /*
! * grottiness needed for old COPY OUT protocol
! */
pq_startcopyout();
copy_dest = COPY_OLD_FE;
}
***************
*** 226,232 ****
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /* new way */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
--- 275,283 ----
{
if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 3)
{
! /*
! * new way
! */
StringInfoData buf;
int16 format = (binary ? 1 : 0);
int i;
***************
*** 242,266 ****
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /* old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
else
{
! /* very old way */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
! /* We *must* flush here to ensure FE knows it can send. */
pq_flush();
}
--- 293,325 ----
}
else if (PG_PROTOCOL_MAJOR(FrontendProtocol) >= 2)
{
! /*
! * old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
else
{
! /*
! * very old way
! */
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
! /*
! * We *must* flush here to ensure FE knows it can send.
! */
pq_flush();
}
***************
*** 271,290 ****
{
if (binary)
{
! /* Need to flush out file trailer word */
CopySendEndOfRow(true);
}
else
{
! /* Shouldn't have any unsent data */
Assert(copy_msgbuf->len == 0);
}
! /* Send Copy Done message */
pq_putemptymessage('c');
}
else
{
! /* The FE/BE protocol uses \n as newline for all platforms */
CopySendData("\\.\n", 3);
pq_endcopyout(false);
}
--- 330,357 ----
{
if (binary)
{
! /*
! * Need to flush out file trailer word
! */
CopySendEndOfRow(true);
}
else
{
! /*
! * Shouldn't have any unsent data
! */
Assert(copy_msgbuf->len == 0);
}
! /*
! * Send Copy Done message
! */
pq_putemptymessage('c');
}
else
{
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
CopySendData("\\.\n", 3);
pq_endcopyout(false);
}
***************
*** 314,323 ****
case COPY_OLD_FE:
if (pq_putbytes((char *) databuf, datasize))
{
! /* no hope of recovering connection sync, so FATAL */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
--- 381,392 ----
case COPY_OLD_FE:
if (pq_putbytes((char *) databuf, datasize))
{
! /*
! * no hope of recovering connection sync, so FATAL
! */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
***************
*** 346,352 ****
case COPY_FILE:
if (!binary)
{
! /* Default line termination depends on platform */
#ifndef WIN32
CopySendChar('\n');
#else
--- 415,423 ----
case COPY_FILE:
if (!binary)
{
! /*
! * Default line termination depends on platform
! */
#ifndef WIN32
CopySendChar('\n');
#else
***************
*** 355,371 ****
}
break;
case COPY_OLD_FE:
! /* The FE/BE protocol uses \n as newline for all platforms */
if (!binary)
CopySendChar('\n');
break;
case COPY_NEW_FE:
! /* The FE/BE protocol uses \n as newline for all platforms */
if (!binary)
CopySendChar('\n');
! /* Dump the accumulated row as one CopyData message */
(void) pq_putmessage('d', copy_msgbuf->data, copy_msgbuf->len);
! /* Reset copy_msgbuf to empty */
copy_msgbuf->len = 0;
copy_msgbuf->data[0] = '\0';
break;
--- 426,450 ----
}
break;
case COPY_OLD_FE:
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
if (!binary)
CopySendChar('\n');
break;
case COPY_NEW_FE:
! /*
! * The FE/BE protocol uses \n as newline for all platforms
! */
if (!binary)
CopySendChar('\n');
! /*
! * Dump the accumulated row as one CopyData message
! */
(void) pq_putmessage('d', copy_msgbuf->data, copy_msgbuf->len);
! /*
! * Reset copy_msgbuf to empty
! */
copy_msgbuf->len = 0;
copy_msgbuf->data[0] = '\0';
break;
***************
*** 383,407 ****
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
*/
! static void
CopyGetData(void *databuf, int datasize)
{
switch (copy_dest)
{
case COPY_FILE:
! fread(databuf, datasize, 1, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
case COPY_OLD_FE:
if (pq_getbytes((char *) databuf, datasize))
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
--- 462,494 ----
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
+ *
+ * Returns: the number of bytes that were successfully read
+ * into the data buffer.
*/
! static int
CopyGetData(void *databuf, int datasize)
{
+ size_t bytesread = 0;
+
switch (copy_dest)
{
case COPY_FILE:
! bytesread = fread(databuf, 1, datasize, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
case COPY_OLD_FE:
if (pq_getbytes((char *) databuf, datasize))
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
+ bytesread += datasize; /* update the count of bytes that were read so far */
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
***************
*** 410,441 ****
while (copy_msgbuf->cursor >= copy_msgbuf->len)
{
! /* Try to receive another message */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
break;
case 'c': /* CopyDone */
! /* COPY IN correctly terminated by frontend */
fe_eof = true;
! return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
--- 497,534 ----
while (copy_msgbuf->cursor >= copy_msgbuf->len)
{
! /*
! * Try to receive another message
! */
int mtype;
! readmessage:
mtype = pq_getbyte();
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg
! ("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg
! ("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
break;
case 'c': /* CopyDone */
! /*
! * COPY IN correctly terminated by frontend
! */
fe_eof = true;
! return bytesread;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
***************
*** 450,457 ****
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
! errmsg("unexpected message type 0x%02X during COPY from stdin",
! mtype)));
break;
}
}
--- 543,551 ----
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
! errmsg
! ("unexpected message type 0x%02X during COPY from stdin",
! mtype)));
break;
}
}
***************
*** 460,469 ****
--- 554,566 ----
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
+ bytesread += avail; /* update the count of bytes that were read so far */
datasize -= avail;
}
break;
}
+
+ return bytesread;
}
static int
***************
*** 480,486 ****
ch = pq_getbyte();
if (ch == EOF)
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
--- 577,585 ----
ch = pq_getbyte();
if (ch == EOF)
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
***************
*** 529,535 ****
ch = pq_peekbyte();
if (ch == EOF)
{
! /* Only a \. terminator is legal EOF in old protocol */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
--- 628,636 ----
ch = pq_peekbyte();
if (ch == EOF)
{
! /*
! * Only a \. terminator is legal EOF in old protocol
! */
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
***************
*** 565,579 ****
case COPY_FILE:
if (!pickup)
{
! /* We don't want to pick it up - so put it back in there */
ungetc(c, copy_file);
}
! /* If we wanted to pick it up, it's already done */
break;
case COPY_OLD_FE:
if (pickup)
{
! /* We want to pick it up */
(void) pq_getbyte();
}
--- 666,686 ----
case COPY_FILE:
if (!pickup)
{
! /*
! * We don't want to pick it up - so put it back in there
! */
ungetc(c, copy_file);
}
! /*
! * If we wanted to pick it up, it's already done
! */
break;
case COPY_OLD_FE:
if (pickup)
{
! /*
! * We want to pick it up
! */
(void) pq_getbyte();
}
***************
*** 585,594 ****
case COPY_NEW_FE:
if (!pickup)
{
! /* We don't want to pick it up - so put it back in there */
copy_msgbuf->cursor--;
}
! /* If we wanted to pick it up, it's already done */
break;
}
}
--- 692,705 ----
case COPY_NEW_FE:
if (!pickup)
{
! /*
! * We don't want to pick it up - so put it back in there
! */
copy_msgbuf->cursor--;
}
! /*
! * If we wanted to pick it up, it's already done
! */
break;
}
}
***************
*** 695,701 ****
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
--- 806,812 ----
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
***************
*** 708,714 ****
AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
AclResult aclresult;
! /* Extract options from the statement node tree */
foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
--- 819,827 ----
AclMode required_access = (is_from ? ACL_INSERT : ACL_SELECT);
AclResult aclresult;
! /*
! * Extract options from the statement node tree
! */
foreach(option, stmt->options)
{
DefElem *defel = (DefElem *) lfirst(option);
***************
*** 794,801 ****
force_notnull = (List *) defel->arg;
}
else
! elog(ERROR, "option \"%s\" not recognized",
! defel->defname);
}
if (binary && delim)
--- 907,913 ----
force_notnull = (List *) defel->arg;
}
else
! elog(ERROR, "option \"%s\" not recognized", defel->defname);
}
if (binary && delim)
***************
*** 813,819 ****
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify NULL in BINARY mode")));
! /* Set defaults */
if (!delim)
delim = csv_mode ? "," : "\t";
--- 925,933 ----
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify NULL in BINARY mode")));
! /*
! * Set defaults
! */
if (!delim)
delim = csv_mode ? "," : "\t";
***************
*** 828,846 ****
escape = quote;
}
! /* Only single-character delimiter strings are supported. */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /* Check header */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY HEADER available only in CSV mode")));
! /* Check quote */
if (!csv_mode && quote != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 942,968 ----
escape = quote;
}
! escapec = '\\'; /* will be configurable in the future */
!
! /*
! * Only single-character delimiter strings are supported.
! */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /*
! * Check header
! */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY HEADER available only in CSV mode")));
! /*
! * Check quote
! */
if (!csv_mode && quote != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 851,857 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote must be a single character")));
! /* Check escape */
if (!csv_mode && escape != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 973,981 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY quote must be a single character")));
! /*
! * Check escape
! */
if (!csv_mode && escape != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 862,868 ****
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape must be a single character")));
! /* Check force_quote */
if (!csv_mode && force_quote != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 986,994 ----
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY escape must be a single character")));
! /*
! * Check force_quote
! */
if (!csv_mode && force_quote != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 870,909 ****
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
! /* Check force_notnull */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null only available using COPY FROM")));
! /* Don't allow the delimiter to appear in the null string. */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY delimiter must not appear in the NULL specification")));
! /* Don't allow the csv quote char to appear in the null string. */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("CSV quote character must not appear in the NULL specification")));
! /* Open and lock the relation, using the appropriate lock type. */
! rel = heap_openrv(relation, (is_from ? RowExclusiveLock : AccessShareLock));
! /* check read-only transaction */
! if (XactReadOnly && !is_from && !isTempNamespace(RelationGetNamespace(rel)))
ereport(ERROR,
(errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION),
errmsg("transaction is read-only")));
! /* Check permissions. */
aclresult = pg_class_aclcheck(RelationGetRelid(rel), GetUserId(),
required_access);
if (aclresult != ACLCHECK_OK)
--- 996,1052 ----
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
! /*
! * Check force_notnull
! */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY force not null only available using COPY FROM")));
! /*
! * Don't allow the delimiter to appear in the null string.
! */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("COPY delimiter must not appear in the NULL specification")));
! /*
! * Don't allow the csv quote char to appear in the null string.
! */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg
! ("CSV quote character must not appear in the NULL specification")));
! /*
! * Open and lock the relation, using the appropriate lock type.
! */
! rel =
! heap_openrv(relation, (is_from ? RowExclusiveLock : AccessShareLock));
! /*
! * check read-only transaction
! */
! if (XactReadOnly && !is_from
! && !isTempNamespace(RelationGetNamespace(rel)))
ereport(ERROR,
(errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION),
errmsg("transaction is read-only")));
! /*
! * Check permissions.
! */
aclresult = pg_class_aclcheck(RelationGetRelid(rel), GetUserId(),
required_access);
if (aclresult != ACLCHECK_OK)
***************
*** 914,932 ****
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
! /* Don't allow COPY w/ OIDs to or from a table without them */
if (oids && !rel->rd_rel->relhasoids)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("table \"%s\" does not have OIDs",
RelationGetRelationName(rel))));
! /* Generate or convert list of attributes to process */
attnumlist = CopyGetAttnums(rel, attnamelist);
! /* Check that FORCE QUOTE references valid COPY columns */
if (force_quote)
{
TupleDesc tupDesc = RelationGetDescr(rel);
--- 1057,1081 ----
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
! /*
! * Don't allow COPY w/ OIDs to or from a table without them
! */
if (oids && !rel->rd_rel->relhasoids)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_COLUMN),
errmsg("table \"%s\" does not have OIDs",
RelationGetRelationName(rel))));
! /*
! * Generate or convert list of attributes to process
! */
attnumlist = CopyGetAttnums(rel, attnamelist);
! /*
! * Check that FORCE QUOTE references valid COPY columns
! */
if (force_quote)
{
TupleDesc tupDesc = RelationGetDescr(rel);
***************
*** 942,953 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /* Check that FORCE NOT NULL references valid COPY columns */
if (force_notnull)
{
ListCell *cur;
--- 1091,1105 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg
! ("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /*
! * Check that FORCE NOT NULL references valid COPY columns
! */
if (force_notnull)
{
ListCell *cur;
***************
*** 963,981 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
--- 1115,1154 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg
! ("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
! /*
! * Set up variables to avoid per-attribute overhead.
! */
! initStringInfo(&attr_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
+ /*
+ * check if the client encoding is one of the 5 encodings
+ * that are not supported as a server encodings.
+ */
+ switch (client_encoding)
+ {
+ case PG_SJIS:
+ case PG_BIG5:
+ case PG_GBK:
+ case PG_UHC:
+ case PG_GB18030:
+ client_encoding_only = true;
+ break;
+ default:
+ client_encoding_only = false;
+ }
+
+
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
***************
*** 998,1005 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1171,1178 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1017,1024 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1190,1197 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1029,1036 ****
errmsg("\"%s\" is a directory", filename)));
}
}
! CopyFrom(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
! quote, escape, force_notnull_atts, header_line);
}
else
{ /* copy from database to file */
--- 1202,1218 ----
errmsg("\"%s\" is a directory", filename)));
}
}
!
! if (csv_mode || binary) /* old path */
! CopyFrom(rel, attnumlist, binary, oids, delim, null_print,
! csv_mode, quote, escape, force_notnull_atts, header_line);
! /*
! * new path for improved performance (only for delimited format for now)
! */
! else
! CopyFromDelimited(rel, attnumlist, binary, oids, delim, null_print,
! csv_mode, quote, escape, force_notnull_atts);
!
}
else
{ /* copy from database to file */
***************
*** 1049,1056 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1231,1238 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1071,1077 ****
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
--- 1253,1260 ----
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg
! ("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
***************
*** 1080,1087 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1263,1270 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1099,1114 ****
if (!pipe)
{
! /* we assume only the write case could fail here */
if (FreeFile(copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m",
! filename)));
}
! pfree(attribute_buf.data);
pfree(line_buf.data);
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
--- 1282,1299 ----
if (!pipe)
{
! /*
! * we assume only the write case could fail here
! */
if (FreeFile(copy_file))
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not write to file \"%s\": %m", filename)));
}
! pfree(attr_buf.data);
pfree(line_buf.data);
+
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
***************
*** 1178,1184 ****
num_phys_attrs = tupDesc->natts;
attr_count = list_length(attnumlist);
! /* Get info about the columns we need to process. */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
foreach(cur, attnumlist)
--- 1363,1371 ----
num_phys_attrs = tupDesc->natts;
attr_count = list_length(attnumlist);
! /*
! * Get info about the columns we need to process.
! */
out_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
force_quote = (bool *) palloc(num_phys_attrs * sizeof(bool));
foreach(cur, attnumlist)
***************
*** 1189,1200 ****
if (binary)
getTypeBinaryOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid,
! &isvarlena);
else
getTypeOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid,
! &isvarlena);
fmgr_info(out_func_oid, &out_functions[attnum - 1]);
if (list_member_int(force_quote_atts, attnum))
--- 1376,1385 ----
if (binary)
getTypeBinaryOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid, &isvarlena);
else
getTypeOutputInfo(attr[attnum - 1]->atttypid,
! &out_func_oid, &isvarlena);
fmgr_info(out_func_oid, &out_functions[attnum - 1]);
if (list_member_int(force_quote_atts, attnum))
***************
*** 1217,1233 ****
if (binary)
{
! /* Generate header for a binary copy */
int32 tmp;
! /* Signature */
CopySendData((char *) BinarySignature, 11);
! /* Flags field */
tmp = 0;
if (oids)
tmp |= (1 << 16);
CopySendInt32(tmp);
! /* No header extension */
tmp = 0;
CopySendInt32(tmp);
}
--- 1402,1426 ----
if (binary)
{
! /*
! * Generate header for a binary copy
! */
int32 tmp;
! /*
! * Signature
! */
CopySendData((char *) BinarySignature, 11);
! /*
! * Flags field
! */
tmp = 0;
if (oids)
tmp |= (1 << 16);
CopySendInt32(tmp);
! /*
! * No header extension
! */
tmp = 0;
CopySendInt32(tmp);
}
***************
*** 1242,1253 ****
pg_server_to_client((unsigned char *) null_print,
strlen(null_print));
! /* if a header has been requested send the line */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 1435,1448 ----
pg_server_to_client((unsigned char *) null_print,
strlen(null_print));
! /*
! * if a header has been requested send the line
! */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1280,1304 ****
if (binary)
{
! /* Binary per-tuple header */
CopySendInt16(attr_count);
! /* Send OID if wanted --- note attr_count doesn't include it */
if (oids)
{
Oid oid = HeapTupleGetOid(tuple);
! /* Hack --- assume Oid is same size as int32 */
CopySendInt32(sizeof(int32));
CopySendInt32(oid);
}
}
else
{
! /* Text format has no per-tuple header, but send OID if wanted */
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum(HeapTupleGetOid(tuple))));
CopySendString(string);
need_delim = true;
}
--- 1475,1509 ----
if (binary)
{
! /*
! * Binary per-tuple header
! */
CopySendInt16(attr_count);
! /*
! * Send OID if wanted --- note attr_count doesn't include it
! */
if (oids)
{
Oid oid = HeapTupleGetOid(tuple);
! /*
! * Hack --- assume Oid is same size as int32
! */
CopySendInt32(sizeof(int32));
CopySendInt32(oid);
}
}
else
{
! /*
! * Text format has no per-tuple header, but send OID if wanted
! */
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum
! (HeapTupleGetOid
! (tuple))));
CopySendString(string);
need_delim = true;
}
***************
*** 1330,1342 ****
{
if (!binary)
{
! string = DatumGetCString(FunctionCall1(&out_functions[attnum - 1],
! value));
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
--- 1535,1548 ----
{
if (!binary)
{
! string =
! DatumGetCString(FunctionCall1
! (&out_functions[attnum - 1], value));
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
***************
*** 1346,1354 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
! /* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
VARSIZE(outputbytes) - VARHDRSZ);
--- 1552,1563 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1
! (&out_functions[attnum - 1], value));
! /*
! * We assume the result will not have been toasted
! */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
VARSIZE(outputbytes) - VARHDRSZ);
***************
*** 1365,1371 ****
if (binary)
{
! /* Generate trailer for a binary copy */
CopySendInt16(-1);
}
--- 1574,1582 ----
if (binary)
{
! /*
! * Generate trailer for a binary copy
! */
CopySendInt16(-1);
}
***************
*** 1384,1390 ****
{
if (copy_binary)
{
! /* can't usefully display the data */
if (copy_attname)
errcontext("COPY %s, line %d, column %s",
copy_relname, copy_lineno, copy_attname);
--- 1595,1603 ----
{
if (copy_binary)
{
! /*
! * can't usefully display the data
! */
if (copy_attname)
errcontext("COPY %s, line %d, column %s",
copy_relname, copy_lineno, copy_attname);
***************
*** 1395,1423 ****
{
if (copy_attname)
{
! /* error is relevant to a particular column */
! limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
! copy_relname, copy_lineno, copy_attname,
! attribute_buf.data);
}
else
{
! /* error is relevant to a particular line */
! if (line_buf_converted ||
! client_encoding == server_encoding)
{
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
! copy_relname, copy_lineno,
! line_buf.data);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
--- 1608,1641 ----
{
if (copy_attname)
{
! /*
! * error is relevant to a particular column
! */
! limit_printout_length(&attr_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
! copy_relname, copy_lineno, copy_attname, attr_buf.data);
}
else
{
! /*
! * error is relevant to a particular line
! */
! if (line_buf_converted || client_encoding == server_encoding)
{
+ /*
+ * Strip off the newline
+ */
+ *(line_buf.data + line_buf.len - 1) = '\0';
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
! copy_relname, copy_lineno, line_buf.data);
}
else
{
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
***************
*** 1445,1465 ****
int len;
! /* Fast path if definitely okay */
if (buf->len <= MAX_COPY_DATA_DISPLAY)
return;
! /* Apply encoding-dependent truncation */
len = pg_mbcliplen(buf->data, buf->len, MAX_COPY_DATA_DISPLAY);
if (buf->len <= len)
return; /* no need to truncate */
buf->len = len;
buf->data[len] = '\0';
! /* Add "..." to show we truncated the input */
appendStringInfoString(buf, "...");
}
/*
* Copy FROM file to relation.
*/
--- 1663,1690 ----
int len;
! /*
! * Fast path if definitely okay
! */
if (buf->len <= MAX_COPY_DATA_DISPLAY)
return;
! /*
! * Apply encoding-dependent truncation
! */
len = pg_mbcliplen(buf->data, buf->len, MAX_COPY_DATA_DISPLAY);
if (buf->len <= len)
return; /* no need to truncate */
buf->len = len;
buf->data[len] = '\0';
! /*
! * Add "..." to show we truncated the input
! */
appendStringInfoString(buf, "...");
}
+
/*
* Copy FROM file to relation.
*/
***************
*** 1510,1521 ****
* code here that basically duplicated execUtils.c ...)
*/
resultRelInfo = makeNode(ResultRelInfo);
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
resultRelInfo->ri_RelationDesc = rel;
resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers * sizeof(FmgrInfo));
resultRelInfo->ri_TrigInstrument = NULL;
ExecOpenIndices(resultRelInfo);
--- 1735,1748 ----
* code here that basically duplicated execUtils.c ...)
*/
resultRelInfo = makeNode(ResultRelInfo);
+
resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
resultRelInfo->ri_RelationDesc = rel;
resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
if (resultRelInfo->ri_TrigDesc)
resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers *
! sizeof(FmgrInfo));
resultRelInfo->ri_TrigInstrument = NULL;
ExecOpenIndices(resultRelInfo);
***************
*** 1524,1530 ****
estate->es_num_result_relations = 1;
estate->es_result_relation_info = resultRelInfo;
! /* Set up a tuple slot too */
slot = MakeSingleTupleTableSlot(tupDesc);
econtext = GetPerTupleExprContext(estate);
--- 1751,1759 ----
estate->es_num_result_relations = 1;
estate->es_result_relation_info = resultRelInfo;
! /*
! * Set up a tuple slot too
! */
slot = MakeSingleTupleTableSlot(tupDesc);
econtext = GetPerTupleExprContext(estate);
***************
*** 1539,1557 ****
typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
defmap = (int *) palloc(num_phys_attrs * sizeof(int));
defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs = (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
for (attnum = 1; attnum <= num_phys_attrs; attnum++)
{
! /* We don't need info for dropped attributes */
if (attr[attnum - 1]->attisdropped)
continue;
! /* Fetch the input function and typioparam info */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
--- 1768,1791 ----
typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
defmap = (int *) palloc(num_phys_attrs * sizeof(int));
defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs =
! (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
for (attnum = 1; attnum <= num_phys_attrs; attnum++)
{
! /*
! * We don't need info for dropped attributes
! */
if (attr[attnum - 1]->attisdropped)
continue;
! /*
! * Fetch the input function and typioparam info
! */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
***************
*** 1562,1572 ****
else
force_notnull[attnum - 1] = false;
! /* Get default info if needed */
if (!list_member_int(attnumlist, attnum))
{
! /* attribute is NOT to be copied from input */
! /* use default value if one exists */
Node *defexpr = build_column_default(rel, attnum);
if (defexpr != NULL)
--- 1796,1812 ----
else
force_notnull[attnum - 1] = false;
! /*
! * Get default info if needed
! */
if (!list_member_int(attnumlist, attnum))
{
! /*
! * attribute is NOT to be copied from input
! */
! /*
! * use default value if one exists
! */
Node *defexpr = build_column_default(rel, attnum);
if (defexpr != NULL)
***************
*** 1578,1584 ****
}
}
! /* If it's a domain type, set up to check domain constraints */
if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
{
Param *prm;
--- 1818,1826 ----
}
}
! /*
! * If it's a domain type, set up to check domain constraints
! */
if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
{
Param *prm;
***************
*** 1593,1598 ****
--- 1835,1841 ----
* actual datum during the data input loop.
*/
prm = makeNode(Param);
+
prm->paramkind = PARAM_EXEC;
prm->paramid = 0;
prm->paramtype = getBaseType(attr[attnum - 1]->atttypid);
***************
*** 1608,1614 ****
}
}
! /* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
/*
--- 1851,1859 ----
}
}
! /*
! * Prepare to catch AFTER triggers.
! */
AfterTriggerBeginQuery();
/*
***************
*** 1624,1640 ****
file_has_oids = oids; /* must rely on user to tell us this... */
else
{
! /* Read and verify binary header */
char readSig[11];
int32 tmp;
! /* Signature */
CopyGetData(readSig, 11);
if (CopyGetEof() || memcmp(readSig, BinarySignature, 11) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("COPY file signature not recognized")));
! /* Flags field */
tmp = CopyGetInt32();
if (CopyGetEof())
ereport(ERROR,
--- 1869,1891 ----
file_has_oids = oids; /* must rely on user to tell us this... */
else
{
! /*
! * Read and verify binary header
! */
char readSig[11];
int32 tmp;
! /*
! * Signature
! */
CopyGetData(readSig, 11);
if (CopyGetEof() || memcmp(readSig, BinarySignature, 11) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("COPY file signature not recognized")));
! /*
! * Flags field
! */
tmp = CopyGetInt32();
if (CopyGetEof())
ereport(ERROR,
***************
*** 1645,1684 ****
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
! /* Header extension length */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /* Skip extension header, if present */
while (tmp-- > 0)
{
CopyGetData(readSig, 1);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
if (file_has_oids && binary)
{
! getTypeBinaryInputInfo(OIDOID,
! &in_func_oid, &oid_typioparam);
fmgr_info(in_func_oid, &oid_in_function);
}
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! /* Make room for a PARAM_EXEC value for domain constraint checks */
if (hasConstraints)
econtext->ecxt_param_exec_vals = (ParamExecData *)
palloc0(sizeof(ParamExecData));
! /* Initialize static variables */
fe_eof = false;
eol_type = EOL_UNKNOWN;
copy_binary = binary;
--- 1896,1943 ----
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unrecognized critical flags in COPY file header")));
! /*
! * Header extension length
! */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
! /*
! * Skip extension header, if present
! */
while (tmp-- > 0)
{
CopyGetData(readSig, 1);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
if (file_has_oids && binary)
{
! getTypeBinaryInputInfo(OIDOID, &in_func_oid, &oid_typioparam);
fmgr_info(in_func_oid, &oid_in_function);
}
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! /*
! * Make room for a PARAM_EXEC value for domain constraint checks
! */
if (hasConstraints)
econtext->ecxt_param_exec_vals = (ParamExecData *)
palloc0(sizeof(ParamExecData));
! /*
! * Initialize static variables
! */
fe_eof = false;
eol_type = EOL_UNKNOWN;
copy_binary = binary;
***************
*** 1686,1702 ****
copy_lineno = 0;
copy_attname = NULL;
! /* Set up callback to identify error line number */
errcontext.callback = copy_in_error_callback;
errcontext.arg = NULL;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /* on input just throw the header line away */
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape) ;
}
while (!done)
--- 1945,1965 ----
copy_lineno = 0;
copy_attname = NULL;
! /*
! * Set up callback to identify error line number
! */
errcontext.callback = copy_in_error_callback;
errcontext.arg = NULL;
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /*
! * on input just throw the header line away
! */
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape);
}
while (!done)
***************
*** 1708,1720 ****
copy_lineno++;
! /* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
! /* Switch into its memory context */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
--- 1971,1989 ----
copy_lineno++;
! /*
! * Reset the per-tuple exprcontext
! */
ResetPerTupleExprContext(estate);
! /*
! * Switch into its memory context
! */
MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
! /*
! * Initialize all values for row to NULL
! */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
***************
*** 1724,1731 ****
char *string;
ListCell *cur;
! /* Actually read the line into memory here */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
--- 1993,2002 ----
char *string;
ListCell *cur;
! /*
! * Actually read the line into memory here
! */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
***************
*** 1739,1745 ****
if (file_has_oids)
{
! /* can't be in CSV mode here */
string = CopyReadAttribute(delim, null_print,
&result, &isnull);
--- 2010,2018 ----
if (file_has_oids)
{
! /*
! * can't be in CSV mode here
! */
string = CopyReadAttribute(delim, null_print,
&result, &isnull);
***************
*** 1751,1757 ****
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 2024,2031 ----
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum
! (string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 1760,1766 ****
}
}
! /* Loop to read the user attributes on the line. */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 2034,2042 ----
}
}
! /*
! * Loop to read the user attributes on the line.
! */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1779,1789 ****
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
--- 2055,2065 ----
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
***************
*** 1795,1808 ****
isnull = false;
}
! /* we read an SQL NULL, no need to do anything */
if (!isnull)
{
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
--- 2071,2087 ----
isnull = false;
}
! /*
! * we read an SQL NULL, no need to do anything
! */
if (!isnull)
{
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->
! atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
***************
*** 1818,1828 ****
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
! /* binary */
int16 fld_count;
ListCell *cur;
--- 2097,2109 ----
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
! /*
! * binary
! */
int16 fld_count;
ListCell *cur;
***************
*** 1844,1851 ****
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
--- 2125,2132 ----
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
***************
*** 1864,1871 ****
i++;
values[m] = CopyReadBinaryAttribute(i,
&in_functions[m],
! typioparams[m],
! &isnull);
nulls[m] = isnull ? 'n' : ' ';
copy_attname = NULL;
}
--- 2145,2151 ----
i++;
values[m] = CopyReadBinaryAttribute(i,
&in_functions[m],
! typioparams[m], &isnull);
nulls[m] = isnull ? 'n' : ' ';
copy_attname = NULL;
}
***************
*** 1884,1890 ****
nulls[defmap[i]] = ' ';
}
! /* Next apply any domain constraints */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
--- 2164,2172 ----
nulls[defmap[i]] = ' ';
}
! /*
! * Next apply any domain constraints
! */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
***************
*** 1896,1902 ****
if (exprstate == NULL)
continue; /* no constraint for this attr */
! /* Insert current row's value into the Param value */
prmdata->value = values[i];
prmdata->isnull = (nulls[i] == 'n');
--- 2178,2186 ----
if (exprstate == NULL)
continue; /* no constraint for this attr */
! /*
! * Insert current row's value into the Param value
! */
prmdata->value = values[i];
prmdata->isnull = (nulls[i] == 'n');
***************
*** 1905,1928 ****
* expression to replace the value (consider e.g. a
* timestamp precision restriction).
*/
! values[i] = ExecEvalExpr(exprstate, econtext,
! &isnull, NULL);
nulls[i] = isnull ? 'n' : ' ';
}
}
! /* And now we can form the input tuple. */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /* Triggers and stuff need to be invoked in query context. */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
! /* BEFORE ROW INSERT Triggers */
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
--- 2189,2217 ----
* expression to replace the value (consider e.g. a
* timestamp precision restriction).
*/
! values[i] = ExecEvalExpr(exprstate, econtext, &isnull, NULL);
nulls[i] = isnull ? 'n' : ' ';
}
}
! /*
! * And now we can form the input tuple.
! */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /*
! * Triggers and stuff need to be invoked in query context.
! */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
! /*
! * BEFORE ROW INSERT Triggers
! */
if (resultRelInfo->ri_TrigDesc &&
resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
***************
*** 1941,1973 ****
if (!skip_tuple)
{
! /* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /* Check the constraints of the tuple */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /* OK, store the tuple and create index entries for it */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
! /* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /* Done, clean up */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /* Execute AFTER STATEMENT insertion triggers */
ExecASInsertTriggers(estate, resultRelInfo);
! /* Handle queued AFTER triggers */
AfterTriggerEndQuery(estate);
pfree(values);
--- 2230,2276 ----
if (!skip_tuple)
{
! /*
! * Place tuple in tuple slot
! */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /*
! * Check the constraints of the tuple
! */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /*
! * OK, store the tuple and create index entries for it
! */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
! /*
! * AFTER ROW INSERT Triggers
! */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /*
! * Done, clean up
! */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
ExecASInsertTriggers(estate, resultRelInfo);
! /*
! * Handle queued AFTER triggers
! */
AfterTriggerEndQuery(estate);
pfree(values);
***************
*** 1987,2019 ****
FreeExecutorState(estate);
}
-
/*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
*/
! static bool
! CopyReadLine(char * quote, char * escape)
{
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
unsigned char s[2];
char *cvt;
! bool in_quote = false, last_was_esc = false, csv_mode = false;
! char quotec = '\0', escapec = '\0';
if (quote)
{
csv_mode = true;
quotec = quote[0];
escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
if (quotec == escapec)
escapec = '\0';
}
--- 2290,2852 ----
FreeExecutorState(estate);
}
/*
! * Copy FROM file to relation with faster processing.
*/
! static void
! CopyFromDelimited(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_notnull_atts)
{
! HeapTuple tuple;
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! ExprState **constraintexprs;
! bool *force_notnull;
! bool hasConstraints = false;
! int attnum;
! int i;
! Oid in_func_oid;
! Datum *values;
! char *nulls;
!
! /*
! * an array of offsets into attr_buf that point to beginning of attributes
! */
! int *attr_offsets;
! bool isnull;
! int null_print_len; /* length of null print */
! ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
! TupleTableSlot *slot;
! bool file_has_oids;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
! MemoryContext oldcontext = CurrentMemoryContext;
! ErrorContextCallback errcontext;
! bool no_more_data;
! CopyReadResult result;
! ListCell *cur;
!
!
! tupDesc = RelationGetDescr(rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(attnumlist);
! num_defaults = 0;
!
! /*
! * We need a ResultRelInfo so we can use the regular executor's
! * index-entry-making machinery. (There used to be a huge amount of
! * code here that basically duplicated execUtils.c ...)
! */
! resultRelInfo = makeNode(ResultRelInfo);
!
! resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
! resultRelInfo->ri_RelationDesc = rel;
! resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
! if (resultRelInfo->ri_TrigDesc)
! resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers *
! sizeof(FmgrInfo));
! resultRelInfo->ri_TrigInstrument = NULL;
!
! ExecOpenIndices(resultRelInfo);
!
! estate->es_result_relations = resultRelInfo;
! estate->es_num_result_relations = 1;
! estate->es_result_relation_info = resultRelInfo;
!
! /*
! * Set up a tuple slot too
! */
! slot = MakeSingleTupleTableSlot(tupDesc);
!
! econtext = GetPerTupleExprContext(estate);
!
! /*
! * Pick up the required catalog information for each attribute in the
! * relation, including the input function, the element type (to pass
! * to the input function), and info about defaults and constraints.
! * (Which input function we use depends on text/binary format choice.)
! */
! in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
! typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
! defmap = (int *) palloc(num_phys_attrs * sizeof(int));
! defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs =
! (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
! force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
!
! for (attnum = 1; attnum <= num_phys_attrs; attnum++)
! {
! /*
! * We don't need info for dropped attributes
! */
! if (attr[attnum - 1]->attisdropped)
! continue;
!
! getTypeInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
!
! fmgr_info(in_func_oid, &in_functions[attnum - 1]);
!
! if (list_member_int(force_notnull_atts, attnum))
! force_notnull[attnum - 1] = true;
! else
! force_notnull[attnum - 1] = false;
!
! /*
! * Get default info if needed
! */
! if (!list_member_int(attnumlist, attnum))
! {
! /*
! * attribute is NOT to be copied from input
! */
! /*
! * use default value if one exists
! */
! Node *defexpr = build_column_default(rel, attnum);
!
! if (defexpr != NULL)
! {
! defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
! estate);
! defmap[num_defaults] = attnum - 1;
! num_defaults++;
! }
! }
!
! /*
! * If it's a domain type, set up to check domain constraints
! */
! if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
! {
! Param *prm;
! Node *node;
!
! /*
! * Easiest way to do this is to use parse_coerce.c to set up
! * an expression that checks the constraints. (At present,
! * the expression might contain a length-coercion-function
! * call and/or CoerceToDomain nodes.) The bottom of the
! * expression is a Param node so that we can fill in the
! * actual datum during the data input loop.
! */
! prm = makeNode(Param);
!
! prm->paramkind = PARAM_EXEC;
! prm->paramid = 0;
! prm->paramtype = getBaseType(attr[attnum - 1]->atttypid);
!
! node = coerce_to_domain((Node *) prm,
! prm->paramtype,
! attr[attnum - 1]->atttypid,
! COERCE_IMPLICIT_CAST, false, false);
!
! constraintexprs[attnum - 1] =
! ExecPrepareExpr((Expr *) node, estate);
! hasConstraints = true;
! }
! }
!
! /*
! * Prepare to catch AFTER triggers.
! */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether
! * we should do this for COPY, since it's not really an "INSERT"
! * statement as such. However, executing these triggers maintains
! * consistency with the EACH ROW triggers that we already fire on
! * COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! file_has_oids = oids; /* must rely on user to tell us this... */
!
! values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
! nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! attr_offsets = (int *) palloc(num_phys_attrs * sizeof(int));
!
! /*
! * Make room for a PARAM_EXEC value for domain constraint checks
! */
! if (hasConstraints)
! econtext->ecxt_param_exec_vals = (ParamExecData *)
! palloc0(sizeof(ParamExecData));
!
! /*
! * Initialize static variables
! */
! fe_eof = false;
! eol_type = EOL_UNKNOWN;
! copy_binary = binary;
! copy_relname = RelationGetRelationName(rel);
! copy_lineno = 0;
! copy_attname = NULL;
! line_buf_converted = false;
!
! /*
! * it is MUCH faster to do this once here than in the attribute parse loop
! */
! null_print_len = strlen(null_print);
!
! /*
! * Set up callback to identify error line number
! */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = NULL;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! /*
! * Set up data buffer to hold a chunk of data
! */
! MemSet(input_buf, ' ', COPY_BUF_SIZE * sizeof(char));
! input_buf[COPY_BUF_SIZE] = '\0';
!
! no_more_data = false; /* no more input data to read from file or FE */
! line_done = true;
!
! do
! {
! /*
! * read a chunk of data into the buffer
! */
! size_t bytesread = CopyGetData(input_buf, COPY_BUF_SIZE);
!
! buf_done = false;
!
! /*
! * set buffer pointers to beginning of the buffer
! */
! begloc = input_buf;
! buffer_index = 0;
!
! /*
! * continue if some bytes were read or if we didn't reach EOF. if we both *
! * * reached EOF _and_ no bytes were read, quit the loop we are done
! */
! if (bytesread > 0 || !fe_eof)
! {
!
! while (!buf_done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /*
! * Reset the per-tuple exprcontext
! */
! ResetPerTupleExprContext(estate);
!
! /*
! * Switch into its memory context
! */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! /*
! * Initialize all values for row to NULL
! */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
! /*
! * reset attribute pointers
! */
! MemSet(attr_offsets, 0, num_phys_attrs * sizeof(int));
!
! result = NORMAL_ATTR;
!
! /*
! * Actually read the line into memory here
! */
! line_done = CopyReadLineBuffered(bytesread);
! copy_lineno++;
!
! /*
! * if didn't finish processing data line, skip att parsing and read more data,
! * * unless there is no more data to read... (which means that the original last
! * * data line is missing attrs and we want to catch that error)
! */
! if (!line_done)
! {
! copy_lineno--;
! if (!fe_eof || buf_done)
! break;
! }
!
! if (file_has_oids)
! {
! char *oid_string;
!
! /*
! * can't be in CSV mode here
! */
! oid_string =
! CopyReadOidAttr(delim, null_print, null_print_len,
! &result, &isnull);
!
! if (isnull)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else
! {
! copy_attname = "oid";
! loaded_oid =
! DatumGetObjectId(DirectFunctionCall1
! (oidin,
! CStringGetDatum(oid_string)));
! if (loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! copy_attname = NULL;
! }
! }
!
! /*
! * parse all the attribute in the data line
! */
! CopyReadAllAttrs(delim, null_print, null_print_len,
! nulls, attnumlist, attr_offsets,
! num_phys_attrs, attr);
!
! /*
! * Loop to read the user attributes on the line.
! */
! foreach(cur, attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (nulls[m] == ' ')
! isnull = false;
! else
! isnull = true;
!
! /*
! * we read an SQL NULL, no need to do anything
! */
! if (!isnull)
! {
! copy_attname = NameStr(attr[m]->attname);
! values[m] = FunctionCall3(&in_functions[m],
! CStringGetDatum(attr_buf.
! data +
! attr_offsets
! [m]),
! ObjectIdGetDatum(typioparams
! [m]),
! Int32GetDatum(attr[m]->
! atttypmod));
! copy_attname = NULL;
! }
! }
!
! /*
! * Now compute and insert any defaults available for the columns
! * not provided by the input data. Anything not processed here or
! * above will remain NULL.
! */
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] =
! ExecEvalExpr(defexprs[i], econtext, &isnull, NULL);
! if (!isnull)
! nulls[defmap[i]] = ' ';
! }
!
! /*
! * Next apply any domain constraints
! */
! if (hasConstraints)
! {
! ParamExecData *prmdata =
! &econtext->ecxt_param_exec_vals[0];
!
! for (i = 0; i < num_phys_attrs; i++)
! {
! ExprState *exprstate = constraintexprs[i];
!
! if (exprstate == NULL)
! continue; /* no constraint for this attr */
!
! /*
! * Insert current row's value into the Param value
! */
! prmdata->value = values[i];
! prmdata->isnull = (nulls[i] == 'n');
!
! /*
! * Execute the constraint expression. Allow the
! * expression to replace the value (consider e.g. a
! * timestamp precision restriction).
! */
! values[i] = ExecEvalExpr(exprstate, econtext,
! &isnull, NULL);
! nulls[i] = isnull ? 'n' : ' ';
! }
! }
!
! /*
! * And now we can form the input tuple.
! */
! tuple = heap_formtuple(tupDesc, values, nulls);
!
! if (oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /*
! * Triggers and stuff need to be invoked in query context.
! */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /*
! * BEFORE ROW INSERT Triggers
! */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->
! n_before_row[TRIGGER_EVENT_INSERT] > 0)
! {
! HeapTuple newtuple;
!
! newtuple =
! ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! /*
! * Place tuple in tuple slot
! */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /*
! * Check the constraints of the tuple
! */
! if (rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /*
! * OK, store the tuple and create index entries for it
! */
! simple_heap_insert(rel, tuple);
!
! if (resultRelInfo->ri_NumIndices > 0)
! ExecInsertIndexTuples(slot, &(tuple->t_self), estate,
! false);
!
! /*
! * AFTER ROW INSERT Triggers
! */
! ExecARInsertTriggers(estate, resultRelInfo, tuple);
! }
!
! line_buf.len = 0; /* we can reset line buffer now. */
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! } /* end while(!buf_done) */
! } /* end if (bytesread > 0 || !fe_eof) */
! else
! /*
! * no bytes read, end of data
! */
! {
! no_more_data = TRUE;
! }
! }
! while (!no_more_data);
!
! /*
! * Done, clean up
! */
! error_context_stack = errcontext.previous;
!
! MemoryContextSwitchTo(oldcontext);
!
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /*
! * Handle queued AFTER triggers
! */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! pfree(attr_offsets);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
! pfree(constraintexprs);
! pfree(force_notnull);
!
! ExecDropSingleTupleTableSlot(slot);
!
! ExecCloseIndices(resultRelInfo);
!
! FreeExecutorState(estate);
! }
!
!
! /*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
! */
! static bool
! CopyReadLine(char *quote, char *escape)
! {
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
unsigned char s[2];
char *cvt;
! bool in_quote = false,
! last_was_esc = false,
! csv_mode = false;
! char quotec = '\0',
! escapec = '\0';
if (quote)
{
csv_mode = true;
quotec = quote[0];
escapec = escape[0];
! /*
! * ignore special escape processing if it's the same as quotec
! */
if (quotec == escapec)
escapec = '\0';
}
***************
*** 2021,2051 ****
s[1] = 0;
! /* reset line_buf to empty */
line_buf.len = 0;
line_buf.data[0] = '\0';
line_buf.cursor = 0;
! /* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /* set default status */
result = false;
/*
* In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
*
* In Text mode, for backwards compatibility we allow
* backslashes to escape newline characters. Backslashes other than
* the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
*
* In CSV mode, CR and NL inside q quoted field are just part of the
* data value and are put in line_buf. We keep just enough state
* to know if we are currently in a quoted field or not.
*
! * These four characters, and only these four, are assumed the same in
* frontend and backend encodings.
*
* We do not assume that second and later bytes of a frontend
--- 2854,2890 ----
s[1] = 0;
! /*
! * reset line_buf to empty
! */
line_buf.len = 0;
line_buf.data[0] = '\0';
line_buf.cursor = 0;
! /*
! * mark that encoding conversion hasn't occurred yet
! */
line_buf_converted = false;
! /*
! * set default status
! */
result = false;
/*
* In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
*
* In Text mode, for backwards compatibility we allow
* backslashes to escape newline characters. Backslashes other than
* the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
*
* In CSV mode, CR and NL inside q quoted field are just part of the
* data value and are put in line_buf. We keep just enough state
* to know if we are currently in a quoted field or not.
*
! * These four characters, and only these four, are assumed the same in
* frontend and backend encodings.
*
* We do not assume that second and later bytes of a frontend
***************
*** 2062,2108 ****
if (csv_mode)
{
! /*
* Dealing with quotes and escapes here is mildly tricky. If the
* quote char is also the escape char, there's no problem - we
* just use the char as a toggle. If they are different, we need
* to ensure that we only take account of an escape inside a quoted
* field and immediately preceding a quote char, and not the
* second in a escape-escape sequence.
! */
if (in_quote && c == escapec)
! last_was_esc = ! last_was_esc;
! if (c == quotec && ! last_was_esc)
! in_quote = ! in_quote;
if (c != escapec)
last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
* the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
}
if (!in_quote && c == '\r')
{
if (eol_type == EOL_NL)
{
! if (! csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
! /* Check for \r\n on first line, _and_ handle \r\n. */
if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
int c2 = CopyPeekChar();
--- 2901,2952 ----
if (csv_mode)
{
! /*
* Dealing with quotes and escapes here is mildly tricky. If the
* quote char is also the escape char, there's no problem - we
* just use the char as a toggle. If they are different, we need
* to ensure that we only take account of an escape inside a quoted
* field and immediately preceding a quote char, and not the
* second in a escape-escape sequence.
! */
if (in_quote && c == escapec)
! last_was_esc = !last_was_esc;
! if (c == quotec && !last_was_esc)
! in_quote = !in_quote;
if (c != escapec)
last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
* the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
}
if (!in_quote && c == '\r')
{
if (eol_type == EOL_NL)
{
! if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("literal carriage return found in data"),
! errhint
! ("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unquoted carriage return found in CSV data"),
! errhint
! ("Use quoted CSV field to represent carriage return.")));
}
! /*
! * Check for \r\n on first line, _and_ handle \r\n.
! */
if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
int c2 = CopyPeekChar();
***************
*** 2114,2132 ****
}
else
{
! /* found \r, but no \n */
if (eol_type == EOL_CRNL)
{
if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
--- 2958,2982 ----
}
else
{
! /*
! * found \r, but no \n
! */
if (eol_type == EOL_CRNL)
{
if (!csv_mode)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("literal carriage return found in data"),
! errhint
! ("Use \"\\r\" to represent carriage return.")));
else
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("unquoted carriage return found in data"),
! errhint
! ("Use quoted CSV field to represent carriage return.")));
}
***************
*** 2153,2160 ****
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
}
eol_type = EOL_NL;
break;
--- 3003,3011 ----
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unquoted newline found in data"),
! errhint
! ("Use quoted CSV field to represent newline.")));
!
}
eol_type = EOL_NL;
break;
***************
*** 2162,2169 ****
if ((line_buf.len == 0 || !csv_mode) && c == '\\')
{
! int c2;
!
if (csv_mode)
c2 = CopyPeekChar();
else
--- 3013,3020 ----
if ((line_buf.len == 0 || !csv_mode) && c == '\\')
{
! int c2;
!
if (csv_mode)
c2 = CopyPeekChar();
else
***************
*** 2179,2185 ****
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
if (eol_type == EOL_CRNL)
{
--- 3030,3036 ----
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
if (eol_type == EOL_CRNL)
{
***************
*** 2187,2193 ****
if (c == '\n')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
if (c != '\r')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 3038,3045 ----
if (c == '\n')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("end-of-copy marker does not match previous newline style")));
if (c != '\r')
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 2203,2209 ****
(eol_type == EOL_CR && c != '\r'))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
/*
* In protocol version 3, we should ignore anything after
--- 3055,3062 ----
(eol_type == EOL_CR && c != '\r'))
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg
! ("end-of-copy marker does not match previous newline style")));
/*
* In protocol version 3, we should ignore anything after
***************
*** 2218,2228 ****
result = true; /* report EOF */
break;
}
!
if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /* not EOF mark, so emit \ and following char literally */
appendStringInfoCharMacro(&line_buf, '\\');
}
--- 3071,3083 ----
result = true; /* report EOF */
break;
}
!
if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /*
! * not EOF mark, so emit \ and following char literally
! */
appendStringInfoCharMacro(&line_buf, '\\');
}
***************
*** 2254,2290 ****
}
} /* end of outer loop */
! /* Done reading the line. Convert it to server encoding. */
! if (change_encoding)
{
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /* transfer converted data back to line_buf */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
}
! /* Now it's safe to use the buffer in error messages */
! line_buf_converted = true;
!
! return result;
}
- /*
- * Return decimal value for a hexadecimal digit
- */
- static
- int GetDecimalFromHex(char hex)
- {
- if (isdigit(hex))
- return hex - '0';
- else
- return tolower(hex) - 'a' + 10;
- }
/*----------
* Read the value of a single attribute, performing de-escaping as needed.
--- 3109,3502 ----
}
} /* end of outer loop */
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (change_encoding)
! {
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /*
! * transfer converted data back to line_buf
! */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
! }
!
! /*
! * Now it's safe to use the buffer in error messages
! */
! line_buf_converted = true;
!
! return result;
! }
!
! /*
! * Return decimal value for a hexadecimal digit
! */
! static int
! GetDecimalFromHex(char hex)
! {
! if (isdigit(hex))
! return hex - '0';
! else
! return tolower(hex) - 'a' + 10;
! }
!
! /*
! * Detected the eol type by looking at the first data row.
! * Possible eol types are NL, CR, or CRNL. If eol type was
! * detected, it is set and a boolean true is returned to
! * indicated detection was successful. If the first data row
! * is longer than the input buffer, we return false and will
! * try again in the next buffer.
! */
! static bool
! DetectLineEnd(size_t bytesread)
! {
! int bytes_remaining = COPY_BUF_SIZE;
! char ch_found; /* the character found in the scan ('\n' or '\r' or escapec) */
! char *start = input_buf;
! char *end;
!
! while (true)
! {
! bytes_remaining = COPY_BUF_SIZE - (input_buf - start);
!
! if (bytes_remaining <= 0)
! return false;
!
! if ((end =
! str2chrlen(start, '\n', '\r', bytes_remaining,
! &ch_found)) == NULL)
! {
! return false;
! }
! else
! {
! if (ch_found == '\n')
! {
! eol_type = EOL_NL;
! eol_ch[0] = '\n';
! eol_ch[1] = '\0';
!
! return true;
! }
! if (ch_found == '\r')
! {
! if (*(end + 1) == '\n')
! {
! eol_type = EOL_CRNL;
! eol_ch[0] = '\r';
! eol_ch[1] = '\n';
! }
! else
! {
! eol_type = EOL_CR;
! eol_ch[0] = '\r';
! eol_ch[1] = '\0';
! }
!
! return true;
! }
! }
! }
!
! }
!
!
! /*
! * Finds the next data line that is in the input buffer and loads it into line_buf.
! * Returns an indication if the line that was read is complete (if an unescaped line-end was
! * encountered). If we reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
! */
! static bool
! CopyReadLineBuffered(size_t bytesread)
! {
! int linesize;
! bool transcode = (client_encoding != server_encoding);
! char *cvt;
! bool end_marker;
!
! /*
! * mark that encoding conversion hasn't occurred yet
! */
! line_buf_converted = false;
!
! /*
! * Detect end of line type if not already detected.
! */
! if (eol_type == EOL_UNKNOWN)
! {
! bool eol_detected = DetectLineEnd(bytesread);
!
! if (!eol_detected)
! {
! /*
! * load entire input buffer into line buf, and quit
! */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! line_done = false;
! buf_done = true;
!
! return line_done;
! }
! }
!
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /*
! * if we started scanning from the 1st byte of the buffer
! */
! if (begloc == input_buf)
! {
! /*
! * and had a CR in last byte of prev buf
! */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence (linefeed)
! */
! if (*begloc == eol_ch[1])
! {
! /*
! * load that one linefeed byte and indicate we are done with the data line
! */
! appendBinaryStringInfo(&line_buf, begloc, 1);
! buffer_index++;
! begloc++;
!
! line_done = true;
! esc_in_prevbuf = false;
! cr_in_prevbuf = false;
!
! return line_done;
! }
! }
!
! cr_in_prevbuf = false;
! }
! }
!
! /*
! * (we need a loop so that if eol_ch is found, but prev ch is backslash, we can search
! * for the next eol_ch)
! */
! while (true)
! {
! /*
! * reached end of buffer
! */
! if ((endloc =
! strchrlen(begloc, eol_ch[0], bytesread - buffer_index)) == NULL)
! {
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
!
! if (line_buf.len > 1)
! {
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == escapec)
! {
! esc_in_prevbuf = true;
!
! if (line_buf.len > 2)
! {
! last_ch--;
! if (*last_ch == escapec)
! esc_in_prevbuf = false;
! }
! }
! else if (*last_ch == '\r')
! {
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
! }
! }
!
! line_done = false;
! buf_done = true;
! break;
! }
! else
! /*
! * found the 1st eol ch in input_buf.
! */
! {
! bool eol_found = true;
! bool eol_escaped = true;
!
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
! if (eol_type == EOL_CRNL)
! {
! /*
! * check if there is a '\n' after the '\r'
! */
! if (*(endloc + 1) == '\n')
! {
! /*
! * this is a line end
! */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
! else
! /*
! * just a CR, not a line end
! */
! eol_found = false;
! }
!
! /*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
! */
! if (buffer_index >= bytesread)
! buf_done = true;
!
! /*
! * Check if the 1st end of line ch is escaped.
! */
! if (endloc != input_buf) /* can we look 1 char back? */
! {
! if (*(endloc - 1) != escapec) /* prev char is not an escape */
! eol_escaped = false;
! else
! /*
! * prev char is an escape
! */
! {
! if (endloc != (input_buf + 1)) /* can we look another char back? */
! {
! /*
! * it's a double escape char, so it's not an escape
! */
! if (*(endloc - 2) == escapec)
! eol_escaped = false;
! /*
! * else it's a single escape char, so EOL is ascaped
! */
! }
! else
! {
! /*
! * we need to check in the last buffer
! */
! if (esc_in_prevbuf) /* double escape char, so not an escape */
! eol_escaped = false;
! }
! }
! }
! else
! /*
! * this eol ch is first ch in buffer, check for escape in prev buf
! */
! {
! if (!esc_in_prevbuf)
! eol_escaped = false;
! }
!
! esc_in_prevbuf = false; /* reset variable */
!
! /*
! * if eol was found, and it isn't escaped, line is done
! */
! if ((eol_escaped == false) && eol_found)
! {
! line_done = true;
! break;
! }
! else
! /*
! * stay in the loop and process some more data.
! */
! line_done = false;
!
! } /* end of found eol_ch */
! }
!
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
! {
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /*
! * transfer converted data back to line_buf
! */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
! }
!
! /*
! * indicate that conversion had occured
! */
! line_buf_converted = true;
!
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
!
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
{
! fe_eof = true;
! /*
! * we don't want to process a \. as data line, want to quit.
! */
! line_done = false;
! buf_done = true;
}
! return line_done;
}
/*----------
* Read the value of a single attribute, performing de-escaping as needed.
***************
*** 2314,2324 ****
int end_cursor;
int input_len;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! /* set default status */
*result = END_OF_LINE;
for (;;)
--- 3526,3541 ----
int end_cursor;
int input_len;
! /*
! * reset attr_buf to empty
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /*
! * set default status
! */
*result = END_OF_LINE;
for (;;)
***************
*** 2347,2353 ****
case '5':
case '6':
case '7':
! /* handle \013 */
{
int val;
--- 3564,3572 ----
case '5':
case '6':
case '7':
! /*
! * handle \013
! */
{
int val;
***************
*** 2374,2387 ****
}
break;
case 'x':
! /* Handle \x3F */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
--- 3593,3608 ----
}
break;
case 'x':
! /*
! * Handle \x3F
! */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
***************
*** 2390,2396 ****
if (isxdigit(hexchar))
{
line_buf.cursor++;
! val = (val << 4) + GetDecimalFromHex(hexchar);
}
}
c = val & 0xff;
--- 3611,3619 ----
if (isxdigit(hexchar))
{
line_buf.cursor++;
! val =
! (val << 4) +
! GetDecimalFromHex(hexchar);
}
}
c = val & 0xff;
***************
*** 2422,2431 ****
*/
}
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
! /* check whether raw input matched null marker */
input_len = end_cursor - start_cursor;
if (input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
--- 3645,3656 ----
*/
}
}
! appendStringInfoCharMacro(&attr_buf, c);
}
! /*
! * check whether raw input matched null marker
! */
input_len = end_cursor - start_cursor;
if (input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
***************
*** 2433,2439 ****
else
*isnull = false;
! return attribute_buf.data;
}
--- 3658,4066 ----
else
*isnull = false;
! return attr_buf.data;
! }
!
! /*
! * Read the first attribute. This is mainly used to maintain support
! * for an OID column. All the rest of the columns will be read at once with
! * CopyReadAllAttrs().
! */
! static char *
! CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
! CopyReadResult *result, bool *isnull)
! {
! char delimc = delim[0];
! char *start_loc = line_buf.data + line_buf.cursor;
! char *end_loc;
! int attr_len = 0;
! int bytes_remaining;
!
! /*
! * reset attribute buf to empty
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! /*
! * set default status
! */
! *result = END_OF_LINE;
!
! /*
! * # of bytes that were not yet processed in this line
! */
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! /*
! * got to end of line
! */
! if ((end_loc = strchrlen(start_loc, delimc, bytes_remaining)) == NULL)
! {
! attr_len = bytes_remaining - 1; /* don't count '\n' in len calculation */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 2; /* skip '\n' and '\0' */
!
! *result = END_OF_LINE;
! }
! else
! /*
! * found a delimiter
! */
! {
! /*
! * (we don't care if delim was preceded with a backslash, because it's an
! * invalid OID anyway)
! */
!
! attr_len = end_loc - start_loc; /* we don't include the delimiter ch */
!
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 1;
!
! *result = NORMAL_ATTR;
! }
!
!
! /*
! * check whether raw input matched null marker
! */
! if (attr_len == null_print_len
! && strncmp(start_loc, null_print, attr_len) == 0)
! *isnull = true;
! else
! *isnull = false;
!
! return attr_buf.data;
! }
!
!
! /*
! * Read all attributes. Attributes are parsed from line_buf and
! * inserted (all at once) to attr_buf, while saving pointers to
! * each attribute's starting position.
! *
! * When this routine finishes execution both the nulls array and
! * the attr_offsets array are updated. The attr_offsets will include
! * the offset from the beginning of the attribute array of which
! * each attribute begins. If a specific attribute is not used for this
! * COPY command (ommitted from the column list), a value of 0 will be assigned.
! * For example: for table foo(a,b,c,d,e) and COPY foo(a,b,e)
! * attr_offsets may look something like this after this routine
! * returns: [0,20,0,0,55]. That means that column "a" value starts
! * at byte offset 0, "b" in 20 and "e" in 55, in attr_buf.
! *
! * In the attribute buffer (attr_buf) each attribute
! * is terminated with a '\0', and therefore by using the attr_offsets
! * array we could point to a beginning of an attribute and have it
! * behave as a C string, much like previously done in COPY.
! *
! * Another aspect to improving performance is reducing the frequency
! * of data load into buffers. The original COPY read attribute code
! * loaded a character at a time. In here we try to load a chunk of data
! * at a time. Usually a chunk will include a full data row
! * (unless we have an escaped delim). That effectively reduces the number of
! * loads by a factor of number of bytes per row. This improves performance
! * greatly, unfortunately it add more complexity to the code.
! *
! * Global participants in parsing logic:
! *
! * line_buf.cursor -- an offset from beginning of the line buffer
! * that indicates where we are about to begin the next scan. Note that
! * if we have WITH OIDS this cursor is already shifted past the first
! * OID attribute.
! *
! * attr_buf.cursor -- an offset from the beginning of the
! * attribute buffer that indicates where the current attribute begins.
! */
! static void
! CopyReadAllAttrs(const char *delim, const char *null_print, int null_print_len,
! char *nulls, List *attnumlist, int *attr_offsets,
! int num_phys_attrs, Form_pg_attribute *attr)
! {
! char delimc = delim[0]; /* delimiter character */
! char *scan_start; /* pointer to line buffer where scan should start. */
! char *scan_end; /* pointer to line buffer where char was found */
! char ch_found; /* the character found in the scan (delimc or escape) */
! int attr_pre_len; /* current attr raw len, before processing escapes */
! int attr_post_len; /* current attr len after escaping */
! int m; /* attribute index being parsed */
! int bytes_remaining; /* num of bytes remaining to be scanned in line buf */
! int chunk_start; /* offset to beginning of line chunk to load */
! int chunk_len; /* length of chunk of data to load to attr buf */
! int oct_val; /* byte value for octal escapes */
! int attnum; /* attribute number being parsed */
! ListCell *cur; /* cursor to attribute list used for this COPY */
! int attribute;
!
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /*
! * cursor is now > 0 if we copy WITH OIDS
! */
! scan_start = line_buf.data + line_buf.cursor;
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! attr_pre_len = 0;
! attr_post_len = 0;
!
! /*
! * Scan through the line buffer to read all attributes data
! */
! while (line_buf.cursor < line_buf.len)
! {
! bytes_remaining = line_buf.len - line_buf.cursor;
! ch_found = '\0';
!
! if ((scan_end =
! str2chrlen(scan_start, delimc, escapec, bytes_remaining,
! &ch_found)) == NULL)
! {
! /*
! * GOT TO END OF LINE BUFFER
! */
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /*
! * don't count eol char(s) in attr and chunk len calculation
! */
! if (eol_type == EOL_CRNL)
! {
! attr_pre_len += bytes_remaining - 2;
! chunk_len = line_buf.len - chunk_start - 2;
! }
! else
! {
! attr_pre_len += bytes_remaining - 1;
! chunk_len = line_buf.len - chunk_start - 1;
! }
!
! /*
! * check if this is a NULL value or data value (assumed NULL)
! */
! if (attr_pre_len == null_print_len
! &&
! strncmp(line_buf.data + line_buf.len - attr_pre_len - 1,
! null_print, attr_pre_len) == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! attr_offsets[m] = attr_buf.cursor;
!
!
! /*
! * load the last chunk, the whole buffer in most cases
! */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start,
! chunk_len);
!
! line_buf.cursor += attr_pre_len + 2; /* skip eol char and '\0' to exit loop */
!
! if (lnext(cur) != NULL)
! {
! /*
! * For an empty data line, the previous COPY code will
! * fail it during the conversion stage. We can fail it here
! * already, but then we will fail the regression tests b/c
! * of a different error message. that's why we return so we
! * can get the same error message that regress expects. ahh...
! * this conditional is unnecessary and should be removed soon.
! */
! if (line_buf.len > 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m + 1]->attname))));
! else
! return;
! }
! }
! else
! /*
! * FOUND A DELIMITER OR ESCAPE
! */
! {
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! if (ch_found == delimc) /* found a delimiter */
! {
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /*
! * (we don't include the delimiter ch in length)
! */
! attr_pre_len += scan_end - scan_start;
! /*
! * (we don't include the delimiter ch in length)
! */
! attr_post_len += scan_end - scan_start;
!
! /*
! * check if this is a null print or data (assumed NULL)
! */
! if (attr_pre_len == null_print_len
! &&
! strncmp(scan_end - attr_pre_len, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /*
! * set the pointer to next attribute position
! */
! attr_offsets[m] = attr_buf.cursor;
!
! /*
! * update buffer cursors to our current location, +1 to skip the delimc
! */
! line_buf.cursor = scan_end - line_buf.data + 1;
! attr_buf.cursor += attr_post_len + 1;
!
! /*
! * prepare scan for next attr
! */
! scan_start = line_buf.data + line_buf.cursor;
! cur = lnext(cur);
! attr_pre_len = 0;
! attr_post_len = 0;
! }
! else
! /*
! * found an escape character
! */
! {
! char nextc = *(scan_end + 1);
! char newc;
! int skip = 2;
!
! chunk_len = (scan_end - line_buf.data) - chunk_start + 1;
!
! /*
! * load a chunk of data
! */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start,
! chunk_len);
!
! switch (nextc)
! {
! case '0':
! case '1':
! case '2':
! case '3':
! case '4':
! case '5':
! case '6':
! case '7':
! oct_val = OCTVALUE(nextc);
! nextc = *(scan_end + 2);
! /*
! * (no need for out bad access check since line if buffered)
! */
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! nextc = *(scan_end + 3);
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! }
! }
! newc = oct_val & 0377; /* the escaped byte value */
! break;
! case 'b':
! newc = '\b';
! break;
! case 'f':
! newc = '\f';
! break;
! case 'n':
! newc = '\n';
! break;
! case 'r':
! newc = '\r';
! break;
! case 't':
! newc = '\t';
! break;
! case 'v':
! newc = '\v';
! break;
! default:
! if (nextc == delimc)
! newc = delimc;
! else if (nextc == escapec)
! newc = escapec;
! else
! /*
! * no escape sequence, take next char literaly
! */
! newc = nextc;
! break;
! }
!
! /*
! * update to current length, add escape and escaped chars
! */
! attr_pre_len += scan_end - scan_start + 2;
! /*
! * update to current length, escaped char
! */
! attr_post_len += scan_end - scan_start + 1;
!
! /*
! * Need to get rid of the escape character. This is done by
! * loading the chunk up to including the escape character
! * into the attribute buffer. Then overwritting the backslash
! * with the escaped sequence or char, and continuing to scan
! * from *after* the char than is after the escape in line buf.
! */
! *(attr_buf.data + attr_buf.len - 1) = newc;
! line_buf.cursor = scan_end - line_buf.data + skip;
! scan_start = scan_end + skip;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! }
!
! } /* end delimiter/backslash */
!
! } /* end line buffer scan. */
!
! /*
! * Replace all delimiters with NULL for string termination.
! * NOTE: only delimiters (NOT necessarily all delimc) are replaced.
! * Example (delimc = '|'):
! * - Before: f 1 | f \| 2 | f 3
! * - After : f 1 \0 f | 2 \0 f 3
! */
! for (attribute = 1; attribute < num_phys_attrs; attribute++)
! {
! if (attr_offsets[attribute] != 0)
! *(attr_buf.data + attr_offsets[attribute] - 1) = '\0';
! }
!
}
***************
*** 2477,2487 ****
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! /* set default status */
*result = END_OF_LINE;
for (;;)
--- 4104,4119 ----
bool in_quote = false;
bool saw_quote = false;
! /*
! * reset attr_buf to empty
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /*
! * set default status
! */
*result = END_OF_LINE;
for (;;)
***************
*** 2491,2504 ****
break;
c = line_buf.data[line_buf.cursor++];
! /* unquoted field delimiter */
if (!in_quote && c == delimc)
{
*result = NORMAL_ATTR;
break;
}
! /* start of quoted field (or part of field) */
if (!in_quote && c == quotec)
{
saw_quote = true;
--- 4123,4140 ----
break;
c = line_buf.data[line_buf.cursor++];
! /*
! * unquoted field delimiter
! */
if (!in_quote && c == delimc)
{
*result = NORMAL_ATTR;
break;
}
! /*
! * start of quoted field (or part of field)
! */
if (!in_quote && c == quotec)
{
saw_quote = true;
***************
*** 2506,2512 ****
continue;
}
! /* escape within a quoted field */
if (in_quote && c == escapec)
{
/*
--- 4142,4150 ----
continue;
}
! /*
! * escape within a quoted field
! */
if (in_quote && c == escapec)
{
/*
***************
*** 2519,2525 ****
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attribute_buf, nextc);
line_buf.cursor++;
continue;
}
--- 4157,4163 ----
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attr_buf, nextc);
line_buf.cursor++;
continue;
}
***************
*** 2536,2548 ****
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
if (in_quote)
*result = UNTERMINATED_FIELD;
! /* check whether raw input matched null marker */
input_len = end_cursor - start_cursor;
if (!saw_quote && input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
--- 4174,4188 ----
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attr_buf, c);
}
if (in_quote)
*result = UNTERMINATED_FIELD;
! /*
! * check whether raw input matched null marker
! */
input_len = end_cursor - start_cursor;
if (!saw_quote && input_len == strlen(null_print) &&
strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
***************
*** 2550,2556 ****
else
*isnull = false;
! return attribute_buf.data;
}
/*
--- 4190,4196 ----
else
*isnull = false;
! return attr_buf.data;
}
/*
***************
*** 2578,2606 ****
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! attribute_buf.cursor = 0;
! enlargeStringInfo(&attribute_buf, fld_size);
! CopyGetData(attribute_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attribute_buf.len = fld_size;
! attribute_buf.data[fld_size] = '\0';
! /* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attribute_buf),
ObjectIdGetDatum(typioparam));
! /* Trouble if it didn't eat the whole buffer */
! if (attribute_buf.cursor != attribute_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
--- 4218,4252 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /*
! * reset attr_buf to empty, and load raw data in it
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! enlargeStringInfo(&attr_buf, fld_size);
! CopyGetData(attr_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attr_buf.len = fld_size;
! attr_buf.data[fld_size] = '\0';
! /*
! * Call the column type's binary input converter
! */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attr_buf),
ObjectIdGetDatum(typioparam));
! /*
! * Trouble if it didn't eat the whole buffer
! */
! if (attr_buf.cursor != attr_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
***************
*** 2668,2674 ****
*/
if (!same_encoding)
{
! /* send additional bytes of the char, if any */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
--- 4314,4322 ----
*/
if (!same_encoding)
{
! /*
! * send additional bytes of the char, if any
! */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
***************
*** 2709,2716 ****
*/
for (test_string = string;
! !use_quote && (c = *test_string) != '\0';
! test_string += mblen)
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
use_quote = true;
--- 4357,4363 ----
*/
for (test_string = string;
! !use_quote && (c = *test_string) != '\0'; test_string += mblen)
{
if (c == delimc || c == quotec || c == '\n' || c == '\r')
use_quote = true;
***************
*** 2732,2738 ****
if (!same_encoding)
{
! /* send additional bytes of the char, if any */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
--- 4379,4387 ----
if (!same_encoding)
{
! /*
! * send additional bytes of the char, if any
! */
mblen = pg_encoding_mblen(client_encoding, string);
for (i = 1; i < mblen; i++)
CopySendChar(string[i]);
***************
*** 2759,2765 ****
if (attnamelist == NIL)
{
! /* Generate default column list */
TupleDesc tupDesc = RelationGetDescr(rel);
Form_pg_attribute *attr = tupDesc->attrs;
int attr_count = tupDesc->natts;
--- 4408,4416 ----
if (attnamelist == NIL)
{
! /*
! * Generate default column list
! */
TupleDesc tupDesc = RelationGetDescr(rel);
Form_pg_attribute *attr = tupDesc->attrs;
int attr_count = tupDesc->natts;
***************
*** 2774,2780 ****
}
else
{
! /* Validate the user-supplied list and extract attnums */
ListCell *l;
foreach(l, attnamelist)
--- 4425,4433 ----
}
else
{
! /*
! * Validate the user-supplied list and extract attnums
! */
ListCell *l;
foreach(l, attnamelist)
***************
*** 2782,2791 ****
char *name = strVal(lfirst(l));
int attnum;
! /* Lookup column name, ereport on failure */
! /* Note we disallow system columns here */
attnum = attnameAttNum(rel, name, false);
! /* Check for duplicates */
if (list_member_int(attnums, attnum))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_COLUMN),
--- 4435,4450 ----
char *name = strVal(lfirst(l));
int attnum;
! /*
! * Lookup column name, ereport on failure
! */
! /*
! * Note we disallow system columns here
! */
attnum = attnameAttNum(rel, name, false);
! /*
! * Check for duplicates
! */
if (list_member_int(attnums, attnum))
ereport(ERROR,
(errcode(ERRCODE_DUPLICATE_COLUMN),
***************
*** 2797,2799 ****
--- 4456,4524 ----
return attnums;
}
+
+ /******************************************************
+ * utility functions, should be placed elsewhere in src
+ ******************************************************/
+
+ /*
+ * These are custom versions of the string function strchr().
+ * As opposed to the original strchr which searches through
+ * a string until the target character is found, or a NULL is
+ * found, this version will not return when a NULL is found.
+ * Instead it will search through a pre-defined length of
+ * bytes and will return only if the target character(s) is reached.
+ *
+ * If our client encoding is not a supported server encoding, we
+ * know that it is not safe to look at each character as trailing
+ * byte in a multibyte character may be a 7-bit ASCII equivalent.
+ * Therefore we use pg_encoding_mblen to skip to the end of the
+ * character.
+ *
+ * parameters:
+ * s - string being searched.
+ * c(n) - char we are searching for.
+ * len - maximum # of bytes to search.
+ *
+ * returns:
+ * pointer to c - if c is located within the string.
+ * NULL - if c was not found in specified length of search. Note:
+ * this DOESN'T mean that a '\0' was reached.
+ */
+ char *
+ strchrlen(const char *s, int c, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, s);
+
+ for (start = s; *s != c && s < start + len; s += mblen)
+ mblen = pg_encoding_mblen(client_encoding, s);
+ }
+ else
+ /*
+ * safe to scroll byte by byte
+ */
+ {
+ for (start = s; *s != c && s < start + len; s++)
+ ;
+ }
+
+ return ((*s == c) ? (char *) s : NULL);
+ }
+
+ char *
+ str2chrlen(const char *s, int c1, int c2, size_t len, char *c_found)
+ {
+ const char *start;
+
+ *c_found = '\0';
+
+ for (start = s; *s != c1 && *s != c2 && s < start + len; s++)
+ ;
+
+ *c_found = *s;
+ return (*c_found != '\0' ? (char *) s : NULL);
+ }
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-06-27 20:20 ` Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2005-06-27 20:20 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Luke Lonergan wrote:
>Andrew,
>
>
>
>>You might like to look at running pgindent (see src/tools/pgindent) over
>>the file before cutting a patch. Since this is usually run over each
>>file just before a release, the only badness should be things from
>>recent patches.
>>
>>
>
>I've attached two patches, one gained from running pgindent against the
>current CVS tip copy.c (:-D) and one gained by running the COPY FROM perf
>improvements through the same. Nifty tool!
>
>Only formatting changes in these.
>
>
>
>
Luke,
Something strange has happened. I suspect that you've inadvertantly used
GNU indent or an unpatched BSD indent. pgindent needs a special patched
BSD indent to work according to the PG standards - see the README
cheers
andrew
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
@ 2005-06-28 04:44 ` Luke Lonergan <llonergan@greenplum.com>
2005-06-28 12:53 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 23:14 ` Re: COPY FROM performance improvements Bruce Momjian <pgman@candle.pha.pa.us>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
0 siblings, 4 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-06-28 04:44 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Andrew,
> Something strange has happened. I suspect that you've inadvertantly used
> GNU indent or an unpatched BSD indent. pgindent needs a special patched
> BSD indent to work according to the PG standards - see the README
OK - phew! I generated new symbols for pgindent and fixed a bug in the awk
scripting within (diff attached) and ran against the CVS tip copy.c and got
only minor changes in formatting that appear to be consistent with the rest
of the code. I pgindent'ed the COPY FROM performance modded code and it
looks good and tests good.
Only formatting changes to the previous patch for copy.c attached.
Patch to update pgindent with new symbols and fix a bug in an awk section
(extra \\ in front of a ')').
- Luke
Attachments: download all (2 files, .tar.gz)
fast_copy_V14_8.1.patch (application/octet-stream, 88.2K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.245
diff -c -r1.245 copy.c
*** src/backend/commands/copy.c 2 Jun 2005 01:21:22 -0000 1.245
--- src/backend/commands/copy.c 28 Jun 2005 04:35:09 -0000
***************
*** 51,56 ****
--- 51,57 ----
#define ISOCTAL(c) (((c) >= '0') && ((c) <= '7'))
#define OCTVALUE(c) ((c) - '0')
+ #define COPY_BUF_SIZE 65536
/*
* Represents the different source/dest cases we need to worry about at
***************
*** 61,67 ****
COPY_FILE, /* to/from file */
COPY_OLD_FE, /* to/from frontend (2.0 protocol) */
COPY_NEW_FE /* to/from frontend (3.0 protocol) */
! } CopyDest;
/*
* State indicator showing what stopped CopyReadAttribute()
--- 62,68 ----
COPY_FILE, /* to/from file */
COPY_OLD_FE, /* to/from frontend (2.0 protocol) */
COPY_NEW_FE /* to/from frontend (3.0 protocol) */
! } CopyDest;
/*
* State indicator showing what stopped CopyReadAttribute()
***************
*** 82,89 ****
EOL_NL,
EOL_CR,
EOL_CRNL
! } EolType;
!
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
--- 83,89 ----
EOL_NL,
EOL_CR,
EOL_CRNL
! } EolType;
static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
***************
*** 92,97 ****
--- 92,101 ----
* never been reentrant...
*/
static CopyDest copy_dest;
+ static int eol_ch[2]; /* The byte values of the 1 or 2 eol bytes */
+ static char escapec; /* escape char for delimited data format. */
+ static bool client_encoding_only; /* true if client encoding is a non
+ * supported server encoding */
static FILE *copy_file; /* used if copy_dest == COPY_FILE */
static StringInfo copy_msgbuf; /* used if copy_dest == COPY_NEW_FE */
static bool fe_eof; /* true if detected end of copy data */
***************
*** 105,144 ****
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attribute_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attribute_buf's data buffer!
*/
! static StringInfoData attribute_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attribute_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf;
static bool line_buf_converted;
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line, bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char * quote, char * escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
--- 109,162 ----
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
+ /* Static variables for buffered input parsing */
+ /* leave room for extra null (to enable use of string functions) */
+ static char input_buf[COPY_BUF_SIZE + 1];
+ static char *begloc;
+ static char *endloc;
+ static bool esc_in_prevbuf; /* backslash was last character of the data
+ * input buffer */
+ static bool cr_in_prevbuf; /* CR was last character of the data input
+ * buffer */
+ static bool line_done; /* finished processing the whole line or
+ * stopped in the middle */
+ static bool buf_done; /* finished processing the current buffer */
+ static int buffer_index; /* input buffer index */
+
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attr_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attr_buf's data buffer!
*/
! static StringInfoData attr_buf; /* still used in CopyFrom() */
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attr_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
! static StringInfoData line_buf; /* still used in CopyFromBinary/CSV() */
static bool line_buf_converted;
/* non-export function prototypes */
static void DoCopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_quote_atts, bool header_line, bool fe_copy);
static void CopyTo(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_quote_atts, bool header_line);
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char *quote, char *escape);
static char *CopyReadAttribute(const char *delim, const char *null_print,
CopyReadResult *result, bool *isnull);
static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
***************
*** 151,156 ****
--- 169,187 ----
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
+ static bool CopyReadLineBuffered(size_t bytesread);
+ static void CopyFromDelimited(Relation rel, List *attnumlist, bool binary, bool oids,
+ char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
+ List *force_notnull_atts);
+ static char *CopyReadAttribute(const char *delim, const char *null_print,
+ CopyReadResult *result, bool *isnull);
+ static void CopyReadAllAttrs(const char *delim, const char *null_print, int null_print_len,
+ char *nulls, List *attnumlist, int *attr_offsets, int num_phys_attrs,
+ Form_pg_attribute *attr);
+ static char *CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
+ CopyReadResult *result, bool *isnull);
+ static bool DetectLineEnd(size_t bytesread);
+
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
***************
*** 160,166 ****
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static void CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
--- 191,197 ----
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static int CopyGetData(void *databuf, int datasize);
static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
***************
*** 171,176 ****
--- 202,210 ----
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
+ /* new parsing utils */
+ static char *strchrlen(const char *s, int c, size_t len);
+ static char *str2chrlen(const char *s, int c1, int c2, size_t len, char *c_found);
/*
* Send copy start/stop messages for frontend copies. These have changed
***************
*** 201,207 ****
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
--- 235,241 ----
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('H');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
***************
*** 213,219 ****
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
--- 247,253 ----
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('B');
/* grottiness needed for old COPY OUT protocol */
pq_startcopyout();
***************
*** 246,252 ****
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
--- 280,286 ----
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('G');
copy_dest = COPY_OLD_FE;
}
***************
*** 256,262 ****
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
--- 290,296 ----
if (binary)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY BINARY is not supported to stdout or from stdin")));
pq_putemptymessage('D');
copy_dest = COPY_OLD_FE;
}
***************
*** 317,323 ****
/* no hope of recovering connection sync, so FATAL */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
--- 351,357 ----
/* no hope of recovering connection sync, so FATAL */
ereport(FATAL,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("connection lost during COPY to stdout")));
}
break;
case COPY_NEW_FE:
***************
*** 383,396 ****
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
*/
! static void
CopyGetData(void *databuf, int datasize)
{
switch (copy_dest)
{
case COPY_FILE:
! fread(databuf, datasize, 1, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
--- 417,435 ----
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
+ *
+ * Returns: the number of bytes that were successfully read
+ * into the data buffer.
*/
! static int
CopyGetData(void *databuf, int datasize)
{
+ size_t bytesread = 0;
+
switch (copy_dest)
{
case COPY_FILE:
! bytesread = fread(databuf, 1, datasize, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
***************
*** 402,407 ****
--- 441,448 ----
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
+ bytesread += datasize; /* update the count of bytes that were
+ * read so far */
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
***************
*** 418,428 ****
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
--- 459,469 ----
if (mtype == EOF)
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
if (pq_getmessage(copy_msgbuf, 0))
ereport(ERROR,
(errcode(ERRCODE_CONNECTION_FAILURE),
! errmsg("unexpected EOF on client connection")));
switch (mtype)
{
case 'd': /* CopyData */
***************
*** 430,441 ****
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
--- 471,482 ----
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return bytesread;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
errmsg("COPY from stdin failed: %s",
! pq_getmsgstring(copy_msgbuf))));
break;
case 'H': /* Flush */
case 'S': /* Sync */
***************
*** 460,469 ****
--- 501,514 ----
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
+ bytesread += avail; /* update the count of bytes that were
+ * read so far */
datasize -= avail;
}
break;
}
+
+ return bytesread;
}
static int
***************
*** 695,701 ****
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
--- 740,746 ----
bool binary = false;
bool oids = false;
bool csv_mode = false;
! bool header_line = false;
char *delim = NULL;
char *quote = NULL;
char *escape = NULL;
***************
*** 828,840 ****
escape = quote;
}
/* Only single-character delimiter strings are supported. */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /* Check header */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
--- 873,887 ----
escape = quote;
}
+ escapec = '\\'; /* will be configurable in the future */
+
/* Only single-character delimiter strings are supported. */
if (strlen(delim) != 1)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("COPY delimiter must be a single character")));
! /* Check header */
if (!csv_mode && header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
***************
*** 870,892 ****
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
/* Check force_notnull */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null only available using COPY FROM")));
/* Don't allow the delimiter to appear in the null string. */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY delimiter must not appear in the NULL specification")));
/* Don't allow the csv quote char to appear in the null string. */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
--- 917,939 ----
if (force_quote != NIL && is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force quote only available using COPY TO")));
/* Check force_notnull */
if (!csv_mode && force_notnull != NIL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null available only in CSV mode")));
if (force_notnull != NIL && !is_from)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY force not null only available using COPY FROM")));
/* Don't allow the delimiter to appear in the null string. */
if (strchr(null_print, delim[0]) != NULL)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("COPY delimiter must not appear in the NULL specification")));
/* Don't allow the csv quote char to appear in the null string. */
if (csv_mode && strchr(null_print, quote[0]) != NULL)
***************
*** 914,920 ****
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
/* Don't allow COPY w/ OIDs to or from a table without them */
if (oids && !rel->rd_rel->relhasoids)
--- 961,967 ----
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to COPY to or from a file"),
errhint("Anyone can COPY to stdout or from stdin. "
! "psql's \\copy command also works for anyone.")));
/* Don't allow COPY w/ OIDs to or from a table without them */
if (oids && !rel->rd_rel->relhasoids)
***************
*** 942,949 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
--- 989,996 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE QUOTE column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
***************
*** 963,981 ****
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
--- 1010,1046 ----
if (!list_member_int(attnumlist, attnum))
ereport(ERROR,
(errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
! errmsg("FORCE NOT NULL column \"%s\" not referenced by COPY",
! NameStr(attr[attnum - 1]->attname))));
}
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attr_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
+ /*
+ * check if the client encoding is one of the 5 encodings
+ * that are not supported as a server encodings.
+ */
+ switch (client_encoding)
+ {
+ case PG_SJIS:
+ case PG_BIG5:
+ case PG_GBK:
+ case PG_UHC:
+ case PG_GB18030:
+ client_encoding_only = true;
+ break;
+ default:
+ client_encoding_only = false;
+ }
+
+
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
***************
*** 998,1005 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1063,1070 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy to non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1017,1024 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1082,1089 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for reading: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1029,1036 ****
errmsg("\"%s\" is a directory", filename)));
}
}
! CopyFrom(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
! quote, escape, force_notnull_atts, header_line);
}
else
{ /* copy from database to file */
--- 1094,1112 ----
errmsg("\"%s\" is a directory", filename)));
}
}
!
! if (csv_mode || binary) /* old path */
! CopyFrom(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
! quote, escape, force_notnull_atts, header_line);
!
! /*
! * new path for improved performance (only for delimited format for
! * now)
! */
! else
! CopyFromDelimited(rel, attnumlist, binary, oids, delim, null_print, csv_mode,
! quote, escape, force_notnull_atts);
!
}
else
{ /* copy from database to file */
***************
*** 1049,1056 ****
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
--- 1125,1132 ----
else
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
! errmsg("cannot copy from non-table relation \"%s\"",
! RelationGetRelationName(rel))));
}
if (pipe)
{
***************
*** 1071,1077 ****
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
--- 1147,1153 ----
if (!is_absolute_path(filename))
ereport(ERROR,
(errcode(ERRCODE_INVALID_NAME),
! errmsg("relative path not allowed for COPY to file")));
oumask = umask((mode_t) 022);
copy_file = AllocateFile(filename, PG_BINARY_W);
***************
*** 1080,1087 ****
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
--- 1156,1163 ----
if (copy_file == NULL)
ereport(ERROR,
(errcode_for_file_access(),
! errmsg("could not open file \"%s\" for writing: %m",
! filename)));
fstat(fileno(copy_file), &st);
if (S_ISDIR(st.st_mode))
***************
*** 1106,1114 ****
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attribute_buf.data);
pfree(line_buf.data);
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
--- 1182,1191 ----
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attr_buf.data);
pfree(line_buf.data);
+
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
***************
*** 1245,1253 ****
/* if a header has been requested send the line */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
--- 1322,1330 ----
/* if a header has been requested send the line */
if (header_line)
{
! bool hdr_delim = false;
! char *colname;
!
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
***************
*** 1298,1304 ****
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum(HeapTupleGetOid(tuple))));
CopySendString(string);
need_delim = true;
}
--- 1375,1381 ----
if (oids)
{
string = DatumGetCString(DirectFunctionCall1(oidout,
! ObjectIdGetDatum(HeapTupleGetOid(tuple))));
CopySendString(string);
need_delim = true;
}
***************
*** 1335,1342 ****
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
--- 1412,1419 ----
if (csv_mode)
{
CopyAttributeOutCSV(string, delim, quote, escape,
! (strcmp(string, null_print) == 0 ||
! force_quote[attnum - 1]));
}
else
CopyAttributeOut(string, delim);
***************
*** 1346,1353 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
--- 1423,1431 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
! value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
CopySendData(VARDATA(outputbytes),
***************
*** 1396,1405 ****
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attribute_buf.data);
}
else
{
--- 1474,1483 ----
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attr_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attr_buf.data);
}
else
{
***************
*** 1407,1412 ****
--- 1485,1492 ----
if (line_buf_converted ||
client_encoding == server_encoding)
{
+ /* Strip off the newline */
+ *(line_buf.data + line_buf.len - 1) = '\0';
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
copy_relname, copy_lineno,
***************
*** 1417,1423 ****
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
--- 1497,1503 ----
/*
* Here, the line buffer is still in a foreign encoding,
* and indeed it's quite likely that the error is precisely
! * a failure to do encoding conversion (ie, bad data). We
* dare not try to convert it, and at present there's no way
* to regurgitate it without conversion. So we have to punt
* and just report the line number.
***************
*** 1460,1465 ****
--- 1540,1546 ----
appendStringInfoString(buf, "...");
}
+
/*
* Copy FROM file to relation.
*/
***************
*** 1551,1557 ****
/* Fetch the input function and typioparam info */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
--- 1632,1638 ----
/* Fetch the input function and typioparam info */
if (binary)
getTypeBinaryInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
else
getTypeInputInfo(attr[attnum - 1]->atttypid,
&in_func_oid, &typioparams[attnum - 1]);
***************
*** 1645,1657 ****
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
/* Header extension length */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
/* Skip extension header, if present */
while (tmp-- > 0)
{
--- 1726,1738 ----
if ((tmp >> 16) != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unrecognized critical flags in COPY file header")));
/* Header extension length */
tmp = CopyGetInt32();
if (CopyGetEof() || tmp < 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (missing length)")));
/* Skip extension header, if present */
while (tmp-- > 0)
{
***************
*** 1659,1665 ****
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
--- 1740,1746 ----
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid COPY file header (wrong length)")));
}
}
***************
*** 1696,1702 ****
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape) ;
}
while (!done)
--- 1777,1783 ----
if (header_line)
{
copy_lineno++;
! done = CopyReadLine(quote, escape);
}
while (!done)
***************
*** 1725,1731 ****
ListCell *cur;
/* Actually read the line into memory here */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
--- 1806,1812 ----
ListCell *cur;
/* Actually read the line into memory here */
! done = csv_mode ?
CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
***************
*** 1751,1757 ****
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 1832,1838 ----
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 1779,1789 ****
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
--- 1860,1870 ----
if (csv_mode)
{
string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
if (result == UNTERMINATED_FIELD)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
}
else
string = CopyReadAttribute(delim, null_print,
***************
*** 1801,1808 ****
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
--- 1882,1889 ----
copy_attname = NameStr(attr[m]->attname);
values[m] = FunctionCall3(&in_functions[m],
CStringGetDatum(string),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->atttypmod));
nulls[m] = ' ';
copy_attname = NULL;
}
***************
*** 1818,1824 ****
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
--- 1899,1905 ----
if (result == NORMAL_ATTR && line_buf.len != 0)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
}
else
{
***************
*** 1844,1851 ****
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
--- 1925,1932 ----
copy_attname = "oid";
loaded_oid =
DatumGetObjectId(CopyReadBinaryAttribute(0,
! &oid_in_function,
! oid_typioparam,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
***************
*** 1924,1930 ****
/* BEFORE ROW INSERT Triggers */
if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
HeapTuple newtuple;
--- 2005,2011 ----
/* BEFORE ROW INSERT Triggers */
if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
{
HeapTuple newtuple;
***************
*** 1987,2173 ****
FreeExecutorState(estate);
}
-
/*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
*/
! static bool
! CopyReadLine(char * quote, char * escape)
{
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
! unsigned char s[2];
! char *cvt;
! bool in_quote = false, last_was_esc = false, csv_mode = false;
! char quotec = '\0', escapec = '\0';
! if (quote)
! {
! csv_mode = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
! s[1] = 0;
! /* reset line_buf to empty */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! /* mark that encoding conversion hasn't occurred yet */
! line_buf_converted = false;
! /* set default status */
! result = false;
/*
! * In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
! *
! * In Text mode, for backwards compatibility we allow
! * backslashes to escape newline characters. Backslashes other than
! * the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
! *
! * In CSV mode, CR and NL inside q quoted field are just part of the
! * data value and are put in line_buf. We keep just enough state
! * to know if we are currently in a quoted field or not.
! *
! * These four characters, and only these four, are assumed the same in
! * frontend and backend encodings.
! *
! * We do not assume that second and later bytes of a frontend
! * multibyte character couldn't look like ASCII characters.
*/
! for (;;)
{
! c = CopyGetChar();
! if (c == EOF)
! {
! result = true;
! break;
! }
! if (csv_mode)
! {
! /*
! * Dealing with quotes and escapes here is mildly tricky. If the
! * quote char is also the escape char, there's no problem - we
! * just use the char as a toggle. If they are different, we need
! * to ensure that we only take account of an escape inside a quoted
! * field and immediately preceding a quote char, and not the
! * second in a escape-escape sequence.
! */
! if (in_quote && c == escapec)
! last_was_esc = ! last_was_esc;
! if (c == quotec && ! last_was_esc)
! in_quote = ! in_quote;
! if (c != escapec)
! last_was_esc = false;
! /*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
! * the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
! }
! if (!in_quote && c == '\r')
{
! if (eol_type == EOL_NL)
! {
! if (! csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
! }
! /* Check for \r\n on first line, _and_ handle \r\n. */
! if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
! {
! int c2 = CopyPeekChar();
!
! if (c2 == '\n')
! {
! CopyDonePeek(c2, true); /* eat newline */
! eol_type = EOL_CRNL;
! }
! else
! {
! /* found \r, but no \n */
! if (eol_type == EOL_CRNL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
!
! }
! /*
! * if we got here, it is the first line and we didn't
! * get \n, so put it back
! */
! CopyDonePeek(c2, false);
! eol_type = EOL_CR;
! }
! }
! break;
! }
! if (!in_quote && c == '\n')
! {
! if (eol_type == EOL_CR || eol_type == EOL_CRNL)
{
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal newline found in data"),
! errhint("Use \"\\n\" to represent newline.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
}
- eol_type = EOL_NL;
- break;
}
! if ((line_buf.len == 0 || !csv_mode) && c == '\\')
{
! int c2;
!
! if (csv_mode)
! c2 = CopyPeekChar();
! else
! c2 = c = CopyGetChar();
if (c2 == EOF)
{
--- 2068,2710 ----
FreeExecutorState(estate);
}
/*
! * Copy FROM file to relation with faster processing.
*/
! static void
! CopyFromDelimited(Relation rel, List *attnumlist, bool binary, bool oids,
! char *delim, char *null_print, bool csv_mode, char *quote,
! char *escape, List *force_notnull_atts)
{
! HeapTuple tuple;
! TupleDesc tupDesc;
! Form_pg_attribute *attr;
! AttrNumber num_phys_attrs,
! attr_count,
! num_defaults;
! FmgrInfo *in_functions;
! Oid *typioparams;
! ExprState **constraintexprs;
! bool *force_notnull;
! bool hasConstraints = false;
! int attnum;
! int i;
! Oid in_func_oid;
! Datum *values;
! char *nulls;
! /* an array of offsets into attr_buf that point to beginning of attributes */
! int *attr_offsets;
! bool isnull;
! int null_print_len; /* length of null print */
! ResultRelInfo *resultRelInfo;
! EState *estate = CreateExecutorState(); /* for ExecConstraints() */
! TupleTableSlot *slot;
! bool file_has_oids;
! int *defmap;
! ExprState **defexprs; /* array of default att expressions */
! ExprContext *econtext; /* used for ExecEvalExpr for default atts */
! MemoryContext oldcontext = CurrentMemoryContext;
! ErrorContextCallback errcontext;
! bool no_more_data;
! CopyReadResult result;
! ListCell *cur;
! tupDesc = RelationGetDescr(rel);
! attr = tupDesc->attrs;
! num_phys_attrs = tupDesc->natts;
! attr_count = list_length(attnumlist);
! num_defaults = 0;
! /*
! * We need a ResultRelInfo so we can use the regular executor's
! * index-entry-making machinery. (There used to be a huge amount of
! * code here that basically duplicated execUtils.c ...)
! */
! resultRelInfo = makeNode(ResultRelInfo);
! resultRelInfo->ri_RangeTableIndex = 1; /* dummy */
! resultRelInfo->ri_RelationDesc = rel;
! resultRelInfo->ri_TrigDesc = CopyTriggerDesc(rel->trigdesc);
! if (resultRelInfo->ri_TrigDesc)
! resultRelInfo->ri_TrigFunctions = (FmgrInfo *)
! palloc0(resultRelInfo->ri_TrigDesc->numtriggers * sizeof(FmgrInfo));
! resultRelInfo->ri_TrigInstrument = NULL;
! ExecOpenIndices(resultRelInfo);
! estate->es_result_relations = resultRelInfo;
! estate->es_num_result_relations = 1;
! estate->es_result_relation_info = resultRelInfo;
!
! /* Set up a tuple slot too */
! slot = MakeSingleTupleTableSlot(tupDesc);
!
! econtext = GetPerTupleExprContext(estate);
/*
! * Pick up the required catalog information for each attribute in the
! * relation, including the input function, the element type (to pass
! * to the input function), and info about defaults and constraints.
! * (Which input function we use depends on text/binary format choice.)
*/
! in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
! typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
! defmap = (int *) palloc(num_phys_attrs * sizeof(int));
! defexprs = (ExprState **) palloc(num_phys_attrs * sizeof(ExprState *));
! constraintexprs = (ExprState **) palloc0(num_phys_attrs * sizeof(ExprState *));
! force_notnull = (bool *) palloc(num_phys_attrs * sizeof(bool));
!
! for (attnum = 1; attnum <= num_phys_attrs; attnum++)
{
! /* We don't need info for dropped attributes */
! if (attr[attnum - 1]->attisdropped)
! continue;
! getTypeInputInfo(attr[attnum - 1]->atttypid,
! &in_func_oid, &typioparams[attnum - 1]);
! fmgr_info(in_func_oid, &in_functions[attnum - 1]);
! if (list_member_int(force_notnull_atts, attnum))
! force_notnull[attnum - 1] = true;
! else
! force_notnull[attnum - 1] = false;
! /* Get default info if needed */
! if (!list_member_int(attnumlist, attnum))
{
! /* attribute is NOT to be copied from input */
! /* use default value if one exists */
! Node *defexpr = build_column_default(rel, attnum);
! if (defexpr != NULL)
{
! defexprs[num_defaults] = ExecPrepareExpr((Expr *) defexpr,
! estate);
! defmap[num_defaults] = attnum - 1;
! num_defaults++;
}
}
! /* If it's a domain type, set up to check domain constraints */
! if (get_typtype(attr[attnum - 1]->atttypid) == 'd')
{
! Param *prm;
! Node *node;
!
! /*
! * Easiest way to do this is to use parse_coerce.c to set up
! * an expression that checks the constraints. (At present,
! * the expression might contain a length-coercion-function
! * call and/or CoerceToDomain nodes.) The bottom of the
! * expression is a Param node so that we can fill in the
! * actual datum during the data input loop.
! */
! prm = makeNode(Param);
! prm->paramkind = PARAM_EXEC;
! prm->paramid = 0;
! prm->paramtype = getBaseType(attr[attnum - 1]->atttypid);
!
! node = coerce_to_domain((Node *) prm,
! prm->paramtype,
! attr[attnum - 1]->atttypid,
! COERCE_IMPLICIT_CAST, false, false);
!
! constraintexprs[attnum - 1] = ExecPrepareExpr((Expr *) node, estate);
! hasConstraints = true;
! }
! }
!
! /*
! * Prepare to catch AFTER triggers.
! */
! AfterTriggerBeginQuery();
!
! /*
! * Check BEFORE STATEMENT insertion triggers. It's debateable whether
! * we should do this for COPY, since it's not really an "INSERT"
! * statement as such. However, executing these triggers maintains
! * consistency with the EACH ROW triggers that we already fire on
! * COPY.
! */
! ExecBSInsertTriggers(estate, resultRelInfo);
!
! file_has_oids = oids; /* must rely on user to tell us this... */
!
! values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
! nulls = (char *) palloc(num_phys_attrs * sizeof(char));
! attr_offsets = (int *) palloc(num_phys_attrs * sizeof(int));
!
! /* Make room for a PARAM_EXEC value for domain constraint checks */
! if (hasConstraints)
! econtext->ecxt_param_exec_vals = (ParamExecData *)
! palloc0(sizeof(ParamExecData));
!
! /* Initialize static variables */
! fe_eof = false;
! eol_type = EOL_UNKNOWN;
! copy_binary = binary;
! copy_relname = RelationGetRelationName(rel);
! copy_lineno = 0;
! copy_attname = NULL;
! line_buf_converted = false;
!
! /* it is MUCH faster to do this once here than in the attribute parse loop */
! null_print_len = strlen(null_print);
!
! /* Set up callback to identify error line number */
! errcontext.callback = copy_in_error_callback;
! errcontext.arg = NULL;
! errcontext.previous = error_context_stack;
! error_context_stack = &errcontext;
!
! /* Set up data buffer to hold a chunk of data */
! MemSet(input_buf, ' ', COPY_BUF_SIZE * sizeof(char));
! input_buf[COPY_BUF_SIZE] = '\0';
!
! no_more_data = false; /* no more input data to read from file or FE */
! line_done = true;
!
! do
! {
! /* read a chunk of data into the buffer */
! size_t bytesread = CopyGetData(input_buf, COPY_BUF_SIZE);
!
! buf_done = false;
!
! /* set buffer pointers to beginning of the buffer */
! begloc = input_buf;
! buffer_index = 0;
!
! /*
! * continue if some bytes were read or if we didn't reach EOF. if we
! * both * reached EOF _and_ no bytes were read, quit the loop we are
! * done
! */
! if (bytesread > 0 || !fe_eof)
! {
!
! while (!buf_done)
! {
! bool skip_tuple;
! Oid loaded_oid = InvalidOid;
!
! CHECK_FOR_INTERRUPTS();
!
! /* Reset the per-tuple exprcontext */
! ResetPerTupleExprContext(estate);
!
! /* Switch into its memory context */
! MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
!
! /* Initialize all values for row to NULL */
! MemSet(values, 0, num_phys_attrs * sizeof(Datum));
! MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
! /* reset attribute pointers */
! MemSet(attr_offsets, 0, num_phys_attrs * sizeof(int));
!
! result = NORMAL_ATTR;
!
! /* Actually read the line into memory here */
! line_done = CopyReadLineBuffered(bytesread);
! copy_lineno++;
!
! /*
! * if didn't finish processing data line, skip att parsing and
! * read more data, unless there is no more data to read...
! * (which means that the original last data line is missing
! * attrs and we want to catch that error)
! */
! if (!line_done)
! {
! copy_lineno--;
! if (!fe_eof || buf_done)
! break;
! }
!
! if (file_has_oids)
! {
! char *oid_string;
!
! /* can't be in CSV mode here */
! oid_string = CopyReadOidAttr(delim, null_print, null_print_len,
! &result, &isnull);
!
! if (isnull)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("null OID in COPY data")));
! else
! {
! copy_attname = "oid";
! loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(oid_string)));
! if (loaded_oid == InvalidOid)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid OID in COPY data")));
! copy_attname = NULL;
! }
! }
!
! /* parse all the attribute in the data line */
! CopyReadAllAttrs(delim, null_print, null_print_len,
! nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
!
! /*
! * Loop to read the user attributes on the line.
! */
! foreach(cur, attnumlist)
! {
! int attnum = lfirst_int(cur);
! int m = attnum - 1;
!
! if (nulls[m] == ' ')
! isnull = false;
! else
! isnull = true;
!
! /* we read an SQL NULL, no need to do anything */
! if (!isnull)
! {
! copy_attname = NameStr(attr[m]->attname);
! values[m] = FunctionCall3(&in_functions[m],
! CStringGetDatum(attr_buf.data + attr_offsets[m]),
! ObjectIdGetDatum(typioparams[m]),
! Int32GetDatum(attr[m]->atttypmod));
! copy_attname = NULL;
! }
! }
!
! /*
! * Now compute and insert any defaults available for the columns
! * not provided by the input data. Anything not processed here or
! * above will remain NULL.
! */
! for (i = 0; i < num_defaults; i++)
! {
! values[defmap[i]] = ExecEvalExpr(defexprs[i], econtext, &isnull, NULL);
! if (!isnull)
! nulls[defmap[i]] = ' ';
! }
!
! /*
! * Next apply any domain constraints
! */
! if (hasConstraints)
! {
! ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
!
! for (i = 0; i < num_phys_attrs; i++)
! {
! ExprState *exprstate = constraintexprs[i];
!
! if (exprstate == NULL)
! continue; /* no constraint for this attr */
!
! /* Insert current row's value into the Param value */
! prmdata->value = values[i];
! prmdata->isnull = (nulls[i] == 'n');
!
! /*
! * Execute the constraint expression. Allow the
! * expression to replace the value (consider e.g. a
! * timestamp precision restriction).
! */
! values[i] = ExecEvalExpr(exprstate, econtext,
! &isnull, NULL);
! nulls[i] = isnull ? 'n' : ' ';
! }
! }
!
! /*
! * And now we can form the input tuple.
! */
! tuple = heap_formtuple(tupDesc, values, nulls);
!
! if (oids && file_has_oids)
! HeapTupleSetOid(tuple, loaded_oid);
!
! /*
! * Triggers and stuff need to be invoked in query context.
! */
! MemoryContextSwitchTo(oldcontext);
!
! skip_tuple = false;
!
! /* BEFORE ROW INSERT Triggers */
! if (resultRelInfo->ri_TrigDesc &&
! resultRelInfo->ri_TrigDesc->n_before_row[TRIGGER_EVENT_INSERT] > 0)
! {
! HeapTuple newtuple;
!
! newtuple = ExecBRInsertTriggers(estate, resultRelInfo, tuple);
!
! if (newtuple == NULL) /* "do nothing" */
! skip_tuple = true;
! else if (newtuple != tuple) /* modified by Trigger(s) */
! {
! heap_freetuple(tuple);
! tuple = newtuple;
! }
! }
!
! if (!skip_tuple)
! {
! /* Place tuple in tuple slot */
! ExecStoreTuple(tuple, slot, InvalidBuffer, false);
!
! /*
! * Check the constraints of the tuple
! */
! if (rel->rd_att->constr)
! ExecConstraints(resultRelInfo, slot, estate);
!
! /*
! * OK, store the tuple and create index entries for it
! */
! simple_heap_insert(rel, tuple);
!
! if (resultRelInfo->ri_NumIndices > 0)
! ExecInsertIndexTuples(slot, &(tuple->t_self), estate, false);
!
! /* AFTER ROW INSERT Triggers */
! ExecARInsertTriggers(estate, resultRelInfo, tuple);
! }
!
! line_buf.len = 0; /* we can reset line buffer now. */
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! } /* end while(!buf_done) */
! } /* end if (bytesread > 0 || !fe_eof) */
! else
! /* no bytes read, end of data */
! {
! no_more_data = TRUE;
! }
! } while (!no_more_data);
!
! /*
! * Done, clean up
! */
! error_context_stack = errcontext.previous;
!
! MemoryContextSwitchTo(oldcontext);
!
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
! ExecASInsertTriggers(estate, resultRelInfo);
!
! /*
! * Handle queued AFTER triggers
! */
! AfterTriggerEndQuery(estate);
!
! pfree(values);
! pfree(nulls);
!
! pfree(attr_offsets);
!
! pfree(in_functions);
! pfree(typioparams);
! pfree(defmap);
! pfree(defexprs);
! pfree(constraintexprs);
! pfree(force_notnull);
!
! ExecDropSingleTupleTableSlot(slot);
!
! ExecCloseIndices(resultRelInfo);
!
! FreeExecutorState(estate);
! }
!
!
! /*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
! */
! static bool
! CopyReadLine(char *quote, char *escape)
! {
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
! unsigned char s[2];
! char *cvt;
! bool in_quote = false,
! last_was_esc = false,
! csv_mode = false;
! char quotec = '\0',
! escapec = '\0';
!
! if (quote)
! {
! csv_mode = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
!
!
! s[1] = 0;
!
! /* reset line_buf to empty */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
!
! /* mark that encoding conversion hasn't occurred yet */
! line_buf_converted = false;
!
! /* set default status */
! result = false;
!
! /*
! * In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
! *
! * In Text mode, for backwards compatibility we allow
! * backslashes to escape newline characters. Backslashes other than
! * the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
! *
! * In CSV mode, CR and NL inside q quoted field are just part of the
! * data value and are put in line_buf. We keep just enough state
! * to know if we are currently in a quoted field or not.
! *
! * These four characters, and only these four, are assumed the same in
! * frontend and backend encodings.
! *
! * We do not assume that second and later bytes of a frontend
! * multibyte character couldn't look like ASCII characters.
! */
! for (;;)
! {
! c = CopyGetChar();
! if (c == EOF)
! {
! result = true;
! break;
! }
!
! if (csv_mode)
! {
! /*
! * Dealing with quotes and escapes here is mildly tricky. If the
! * quote char is also the escape char, there's no problem - we
! * just use the char as a toggle. If they are different, we need
! * to ensure that we only take account of an escape inside a quoted
! * field and immediately preceding a quote char, and not the
! * second in a escape-escape sequence.
! */
!
! if (in_quote && c == escapec)
! last_was_esc = !last_was_esc;
! if (c == quotec && !last_was_esc)
! in_quote = !in_quote;
! if (c != escapec)
! last_was_esc = false;
!
! /*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
! * the best we can do.
! */
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
! }
!
! if (!in_quote && c == '\r')
! {
! if (eol_type == EOL_NL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
! }
! /* Check for \r\n on first line, _and_ handle \r\n. */
! if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
! {
! int c2 = CopyPeekChar();
!
! if (c2 == '\n')
! {
! CopyDonePeek(c2, true); /* eat newline */
! eol_type = EOL_CRNL;
! }
! else
! {
! /* found \r, but no \n */
! if (eol_type == EOL_CRNL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
!
! }
!
! /*
! * if we got here, it is the first line and we didn't
! * get \n, so put it back
! */
! CopyDonePeek(c2, false);
! eol_type = EOL_CR;
! }
! }
! break;
! }
! if (!in_quote && c == '\n')
! {
! if (eol_type == EOL_CR || eol_type == EOL_CRNL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal newline found in data"),
! errhint("Use \"\\n\" to represent newline.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
! }
! eol_type = EOL_NL;
! break;
! }
!
! if ((line_buf.len == 0 || !csv_mode) && c == '\\')
! {
! int c2;
!
! if (csv_mode)
! c2 = CopyPeekChar();
! else
! c2 = c = CopyGetChar();
if (c2 == EOF)
{
***************
*** 2179,2185 ****
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
if (eol_type == EOL_CRNL)
{
--- 2716,2723 ----
if (c2 == '.')
{
if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling
! * GetChar() */
if (eol_type == EOL_CRNL)
{
***************
*** 2206,2261 ****
errmsg("end-of-copy marker does not match previous newline style")));
/*
! * In protocol version 3, we should ignore anything after
! * \. up to the protocol end of copy data. (XXX maybe
! * better not to treat \. as special?)
*/
! if (copy_dest == COPY_NEW_FE)
{
! while (c != EOF)
! c = CopyGetChar();
}
- result = true; /* report EOF */
- break;
}
-
- if (csv_mode)
- CopyDonePeek(c2, false); /* not a dot, so put it back */
- else
- /* not EOF mark, so emit \ and following char literally */
- appendStringInfoCharMacro(&line_buf, '\\');
- }
! appendStringInfoCharMacro(&line_buf, c);
! /*
! * When client encoding != server, must be careful to read the
! * extra bytes of a multibyte character exactly, since the
! * encoding might not ensure they don't look like ASCII. When the
! * encodings are the same, we need not do this, since no server
! * encoding we use has ASCII-like following bytes.
! */
! if (change_encoding)
{
! s[0] = c;
! mblen = pg_encoding_mblen(client_encoding, s);
! for (j = 1; j < mblen; j++)
{
! c = CopyGetChar();
! if (c == EOF)
{
! result = true;
! break;
}
- appendStringInfoCharMacro(&line_buf, c);
}
! if (result)
! break; /* out of outer loop */
}
! } /* end of outer loop */
! /* Done reading the line. Convert it to server encoding. */
! if (change_encoding)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
--- 2744,3100 ----
errmsg("end-of-copy marker does not match previous newline style")));
/*
! * In protocol version 3, we should ignore anything after
! * \. up to the protocol end of copy data. (XXX maybe
! * better not to treat \. as special?)
! */
! if (copy_dest == COPY_NEW_FE)
! {
! while (c != EOF)
! c = CopyGetChar();
! }
! result = true; /* report EOF */
! break;
! }
!
! if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
! else
! /* not EOF mark, so emit \ and following char literally */
! appendStringInfoCharMacro(&line_buf, '\\');
! }
!
! appendStringInfoCharMacro(&line_buf, c);
!
! /*
! * When client encoding != server, must be careful to read the
! * extra bytes of a multibyte character exactly, since the
! * encoding might not ensure they don't look like ASCII. When the
! * encodings are the same, we need not do this, since no server
! * encoding we use has ASCII-like following bytes.
! */
! if (change_encoding)
! {
! s[0] = c;
! mblen = pg_encoding_mblen(client_encoding, s);
! for (j = 1; j < mblen; j++)
! {
! c = CopyGetChar();
! if (c == EOF)
! {
! result = true;
! break;
! }
! appendStringInfoCharMacro(&line_buf, c);
! }
! if (result)
! break; /* out of outer loop */
! }
! } /* end of outer loop */
!
! /* Done reading the line. Convert it to server encoding. */
! if (change_encoding)
! {
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /* transfer converted data back to line_buf */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
! }
!
! /* Now it's safe to use the buffer in error messages */
! line_buf_converted = true;
!
! return result;
! }
!
! /*
! * Return decimal value for a hexadecimal digit
! */
! static
! int
! GetDecimalFromHex(char hex)
! {
! if (isdigit(hex))
! return hex - '0';
! else
! return tolower(hex) - 'a' + 10;
! }
!
! /*
! * Detected the eol type by looking at the first data row.
! * Possible eol types are NL, CR, or CRNL. If eol type was
! * detected, it is set and a boolean true is returned to
! * indicated detection was successful. If the first data row
! * is longer than the input buffer, we return false and will
! * try again in the next buffer.
! */
! static bool
! DetectLineEnd(size_t bytesread)
! {
! int bytes_remaining = COPY_BUF_SIZE;
! char ch_found; /* the character found in the scan ('\n' or
! * '\r' or escapec) */
! char *start = input_buf;
! char *end;
!
! while (true)
! {
! bytes_remaining = COPY_BUF_SIZE - (input_buf - start);
!
! if (bytes_remaining <= 0)
! return false;
!
! if ((end = str2chrlen(start, '\n', '\r', bytes_remaining, &ch_found)) == NULL)
! {
! return false;
! }
! else
! {
! if (ch_found == '\n')
! {
! eol_type = EOL_NL;
! eol_ch[0] = '\n';
! eol_ch[1] = '\0';
!
! return true;
! }
! if (ch_found == '\r')
! {
! if (*(end + 1) == '\n')
! {
! eol_type = EOL_CRNL;
! eol_ch[0] = '\r';
! eol_ch[1] = '\n';
! }
! else
! {
! eol_type = EOL_CR;
! eol_ch[0] = '\r';
! eol_ch[1] = '\0';
! }
!
! return true;
! }
! }
! }
!
! }
!
!
! /*
! * Finds the next data line that is in the input buffer and loads it into line_buf.
! * Returns an indication if the line that was read is complete (if an unescaped line-end was
! * encountered). If we reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
! */
! static bool
! CopyReadLineBuffered(size_t bytesread)
! {
! int linesize;
! bool transcode = (client_encoding != server_encoding);
! char *cvt;
! bool end_marker;
!
! /* mark that encoding conversion hasn't occurred yet */
! line_buf_converted = false;
!
! /*
! * Detect end of line type if not already detected.
! */
! if (eol_type == EOL_UNKNOWN)
! {
! bool eol_detected = DetectLineEnd(bytesread);
!
! if (!eol_detected)
! {
! /* load entire input buffer into line buf, and quit */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! line_done = false;
! buf_done = true;
!
! return line_done;
! }
! }
!
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /* if we started scanning from the 1st byte of the buffer */
! if (begloc == input_buf)
! {
! /* and had a CR in last byte of prev buf */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence
! * (linefeed)
*/
! if (*begloc == eol_ch[1])
{
! /*
! * load that one linefeed byte and indicate we are done
! * with the data line
! */
! appendBinaryStringInfo(&line_buf, begloc, 1);
! buffer_index++;
! begloc++;
!
! line_done = true;
! esc_in_prevbuf = false;
! cr_in_prevbuf = false;
!
! return line_done;
}
}
! cr_in_prevbuf = false;
! }
! }
! /*
! * (we need a loop so that if eol_ch is found, but prev ch is backslash,
! * we can search for the next eol_ch)
! */
! while (true)
! {
! /* reached end of buffer */
! if ((endloc = strchrlen(begloc, eol_ch[0], bytesread - buffer_index)) == NULL)
{
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
!
! if (line_buf.len > 1)
{
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == escapec)
{
! esc_in_prevbuf = true;
!
! if (line_buf.len > 2)
! {
! last_ch--;
! if (*last_ch == escapec)
! esc_in_prevbuf = false;
! }
! }
! else if (*last_ch == '\r')
! {
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
}
}
!
! line_done = false;
! buf_done = true;
! break;
}
! else
! /* found the 1st eol ch in input_buf. */
! {
! bool eol_found = true;
! bool eol_escaped = true;
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
! if (eol_type == EOL_CRNL)
! {
! /* check if there is a '\n' after the '\r' */
! if (*(endloc + 1) == '\n')
! {
! /* this is a line end */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
! else
! /* just a CR, not a line end */
! eol_found = false;
! }
!
! /*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
! */
! if (buffer_index >= bytesread)
! buf_done = true;
!
! /*
! * Check if the 1st end of line ch is escaped.
! */
! if (endloc != input_buf) /* can we look 1 char back? */
! {
! if (*(endloc - 1) != escapec) /* prev char is not an escape */
! eol_escaped = false;
! else
! /* prev char is an escape */
! {
! if (endloc != (input_buf + 1)) /* can we look another
! * char back? */
! {
! /* it's a double escape char, so it's not an escape */
! if (*(endloc - 2) == escapec)
! eol_escaped = false;
! /* else it's a single escape char, so EOL is ascaped */
! }
! else
! {
! /* we need to check in the last buffer */
! if (esc_in_prevbuf) /* double escape char, so not
! * an escape */
! eol_escaped = false;
! }
! }
! }
! else
!
! /*
! * this eol ch is first ch in buffer, check for escape in prev
! * buf
! */
! {
! if (!esc_in_prevbuf)
! eol_escaped = false;
! }
!
! esc_in_prevbuf = false; /* reset variable */
!
! /*
! * if eol was found, and it isn't escaped, line is done
! */
! if ((eol_escaped == false) && eol_found)
! {
! line_done = true;
! break;
! }
! else
! /* stay in the loop and process some more data. */
! line_done = false;
!
! } /* end of found eol_ch */
! }
!
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
***************
*** 2264,2291 ****
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* Now it's safe to use the buffer in error messages */
line_buf_converted = true;
! return result;
! }
! /*
! * Return decimal value for a hexadecimal digit
! */
! static
! int GetDecimalFromHex(char hex)
! {
! if (isdigit(hex))
! return hex - '0';
! else
! return tolower(hex) - 'a' + 10;
}
/*----------
* Read the value of a single attribute, performing de-escaping as needed.
*
--- 3103,3151 ----
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* indicate that conversion had occured */
line_buf_converted = true;
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
! {
! fe_eof = true;
! /* we don't want to process a \. as data line, want to quit. */
! line_done = false;
! buf_done = true;
! }
!
! return line_done;
}
+
/*----------
* Read the value of a single attribute, performing de-escaping as needed.
*
***************
*** 2314,2322 ****
int end_cursor;
int input_len;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
/* set default status */
*result = END_OF_LINE;
--- 3174,3183 ----
int end_cursor;
int input_len;
! /* reset attr_buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
/* set default status */
*result = END_OF_LINE;
***************
*** 2377,2387 ****
/* Handle \x3F */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
--- 3238,3248 ----
/* Handle \x3F */
if (line_buf.cursor < line_buf.len)
{
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
line_buf.cursor++;
if (line_buf.cursor < line_buf.len)
***************
*** 2422,2428 ****
*/
}
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
/* check whether raw input matched null marker */
--- 3283,3289 ----
*/
}
}
! appendStringInfoCharMacro(&attr_buf, c);
}
/* check whether raw input matched null marker */
***************
*** 2433,2439 ****
else
*isnull = false;
! return attribute_buf.data;
}
--- 3294,3662 ----
else
*isnull = false;
! return attr_buf.data;
! }
!
! /*
! * Read the first attribute. This is mainly used to maintain support
! * for an OID column. All the rest of the columns will be read at once with
! * CopyReadAllAttrs().
! */
! static char *
! CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
! CopyReadResult *result, bool *isnull)
! {
! char delimc = delim[0];
! char *start_loc = line_buf.data + line_buf.cursor;
! char *end_loc;
! int attr_len = 0;
! int bytes_remaining;
!
! /* reset attribute buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! /* set default status */
! *result = END_OF_LINE;
!
! /* # of bytes that were not yet processed in this line */
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! /* got to end of line */
! if ((end_loc = strchrlen(start_loc, delimc, bytes_remaining)) == NULL)
! {
! attr_len = bytes_remaining - 1; /* don't count '\n' in len calculation */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 2; /* skip '\n' and '\0' */
!
! *result = END_OF_LINE;
! }
! else
! /* found a delimiter */
! {
! /*
! * (we don't care if delim was preceded with a backslash, because it's
! * an invalid OID anyway)
! */
!
! attr_len = end_loc - start_loc; /* we don't include the delimiter ch */
!
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 1;
!
! *result = NORMAL_ATTR;
! }
!
!
! /* check whether raw input matched null marker */
! if (attr_len == null_print_len && strncmp(start_loc, null_print, attr_len) == 0)
! *isnull = true;
! else
! *isnull = false;
!
! return attr_buf.data;
! }
!
!
! /*
! * Read all attributes. Attributes are parsed from line_buf and
! * inserted (all at once) to attr_buf, while saving pointers to
! * each attribute's starting position.
! *
! * When this routine finishes execution both the nulls array and
! * the attr_offsets array are updated. The attr_offsets will include
! * the offset from the beginning of the attribute array of which
! * each attribute begins. If a specific attribute is not used for this
! * COPY command (ommitted from the column list), a value of 0 will be assigned.
! * For example: for table foo(a,b,c,d,e) and COPY foo(a,b,e)
! * attr_offsets may look something like this after this routine
! * returns: [0,20,0,0,55]. That means that column "a" value starts
! * at byte offset 0, "b" in 20 and "e" in 55, in attr_buf.
! *
! * In the attribute buffer (attr_buf) each attribute
! * is terminated with a '\0', and therefore by using the attr_offsets
! * array we could point to a beginning of an attribute and have it
! * behave as a C string, much like previously done in COPY.
! *
! * Another aspect to improving performance is reducing the frequency
! * of data load into buffers. The original COPY read attribute code
! * loaded a character at a time. In here we try to load a chunk of data
! * at a time. Usually a chunk will include a full data row
! * (unless we have an escaped delim). That effectively reduces the number of
! * loads by a factor of number of bytes per row. This improves performance
! * greatly, unfortunately it add more complexity to the code.
! *
! * Global participants in parsing logic:
! *
! * line_buf.cursor -- an offset from beginning of the line buffer
! * that indicates where we are about to begin the next scan. Note that
! * if we have WITH OIDS this cursor is already shifted past the first
! * OID attribute.
! *
! * attr_buf.cursor -- an offset from the beginning of the
! * attribute buffer that indicates where the current attribute begins.
! */
! static void
! CopyReadAllAttrs(const char *delim, const char *null_print, int null_print_len,
! char *nulls, List *attnumlist, int *attr_offsets,
! int num_phys_attrs, Form_pg_attribute *attr)
! {
! char delimc = delim[0]; /* delimiter character */
! char *scan_start; /* pointer to line buffer where scan should
! * start. */
! char *scan_end; /* pointer to line buffer where char was found */
! char ch_found; /* the character found in the scan (delimc or
! * escape) */
! int attr_pre_len; /* current attr raw len, before processing
! * escapes */
! int attr_post_len; /* current attr len after escaping */
! int m; /* attribute index being parsed */
! int bytes_remaining;/* num of bytes remaining to be scanned in
! * line buf */
! int chunk_start; /* offset to beginning of line chunk to load */
! int chunk_len; /* length of chunk of data to load to attr buf */
! int oct_val; /* byte value for octal escapes */
! int attnum; /* attribute number being parsed */
! ListCell *cur; /* cursor to attribute list used for this COPY */
! int attribute;
!
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /* cursor is now > 0 if we copy WITH OIDS */
! scan_start = line_buf.data + line_buf.cursor;
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! attr_pre_len = 0;
! attr_post_len = 0;
!
! /*
! * Scan through the line buffer to read all attributes data
! */
! while (line_buf.cursor < line_buf.len)
! {
! bytes_remaining = line_buf.len - line_buf.cursor;
! ch_found = '\0';
!
! if ((scan_end = str2chrlen(scan_start, delimc, escapec, bytes_remaining, &ch_found))
! == NULL)
! {
! /* GOT TO END OF LINE BUFFER */
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* don't count eol char(s) in attr and chunk len calculation */
! if (eol_type == EOL_CRNL)
! {
! attr_pre_len += bytes_remaining - 2;
! chunk_len = line_buf.len - chunk_start - 2;
! }
! else
! {
! attr_pre_len += bytes_remaining - 1;
! chunk_len = line_buf.len - chunk_start - 1;
! }
!
! /* check if this is a NULL value or data value (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(line_buf.data + line_buf.len - attr_pre_len - 1, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! attr_offsets[m] = attr_buf.cursor;
!
!
! /* load the last chunk, the whole buffer in most cases */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! line_buf.cursor += attr_pre_len + 2; /* skip eol char and
! * '\0' to exit loop */
!
! if (lnext(cur) != NULL)
! {
! /*
! * For an empty data line, the previous COPY code will
! * fail it during the conversion stage. We can fail it here
! * already, but then we will fail the regression tests b/c
! * of a different error message. that's why we return so we
! * can get the same error message that regress expects. ahh...
! * this conditional is unnecessary and should be removed soon.
! */
! if (line_buf.len > 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m + 1]->attname))));
! else
! return;
! }
! }
! else
! /* FOUND A DELIMITER OR ESCAPE */
! {
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! if (ch_found == delimc) /* found a delimiter */
! {
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* (we don't include the delimiter ch in length) */
! attr_pre_len += scan_end - scan_start;
! /* (we don't include the delimiter ch in length) */
! attr_post_len += scan_end - scan_start;
!
! /* check if this is a null print or data (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(scan_end - attr_pre_len, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* set the pointer to next attribute position */
! attr_offsets[m] = attr_buf.cursor;
!
! /*
! * update buffer cursors to our current location, +1 to skip
! * the delimc
! */
! line_buf.cursor = scan_end - line_buf.data + 1;
! attr_buf.cursor += attr_post_len + 1;
!
! /* prepare scan for next attr */
! scan_start = line_buf.data + line_buf.cursor;
! cur = lnext(cur);
! attr_pre_len = 0;
! attr_post_len = 0;
! }
! else
! /* found an escape character */
! {
! char nextc = *(scan_end + 1);
! char newc;
! int skip = 2;
!
! chunk_len = (scan_end - line_buf.data) - chunk_start + 1;
!
! /* load a chunk of data */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! switch (nextc)
! {
! case '0':
! case '1':
! case '2':
! case '3':
! case '4':
! case '5':
! case '6':
! case '7':
! oct_val = OCTVALUE(nextc);
! nextc = *(scan_end + 2);
!
! /*
! * (no need for out bad access check since line if
! * buffered)
! */
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! nextc = *(scan_end + 3);
! if (ISOCTAL(nextc))
! {
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! }
! }
! newc = oct_val & 0377; /* the escaped byte value */
! break;
! case 'b':
! newc = '\b';
! break;
! case 'f':
! newc = '\f';
! break;
! case 'n':
! newc = '\n';
! break;
! case 'r':
! newc = '\r';
! break;
! case 't':
! newc = '\t';
! break;
! case 'v':
! newc = '\v';
! break;
! default:
! if (nextc == delimc)
! newc = delimc;
! else if (nextc == escapec)
! newc = escapec;
! else
! /* no escape sequence, take next char literaly */
! newc = nextc;
! break;
! }
!
! /* update to current length, add escape and escaped chars */
! attr_pre_len += scan_end - scan_start + 2;
! /* update to current length, escaped char */
! attr_post_len += scan_end - scan_start + 1;
!
! /*
! * Need to get rid of the escape character. This is done by
! * loading the chunk up to including the escape character
! * into the attribute buffer. Then overwritting the backslash
! * with the escaped sequence or char, and continuing to scan
! * from *after* the char than is after the escape in line buf.
! */
! *(attr_buf.data + attr_buf.len - 1) = newc;
! line_buf.cursor = scan_end - line_buf.data + skip;
! scan_start = scan_end + skip;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! }
!
! } /* end delimiter/backslash */
!
! } /* end line buffer scan. */
!
! /*
! * Replace all delimiters with NULL for string termination.
! * NOTE: only delimiters (NOT necessarily all delimc) are replaced.
! * Example (delimc = '|'):
! * - Before: f 1 | f \| 2 | f 3
! * - After : f 1 \0 f | 2 \0 f 3
! */
! for (attribute = 1; attribute < num_phys_attrs; attribute++)
! {
! if (attr_offsets[attribute] != 0)
! *(attr_buf.data + attr_offsets[attribute] - 1) = '\0';
! }
!
}
***************
*** 2477,2485 ****
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
/* set default status */
*result = END_OF_LINE;
--- 3700,3709 ----
bool in_quote = false;
bool saw_quote = false;
! /* reset attr_buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
/* set default status */
*result = END_OF_LINE;
***************
*** 2519,2525 ****
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attribute_buf, nextc);
line_buf.cursor++;
continue;
}
--- 3743,3749 ----
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attr_buf, nextc);
line_buf.cursor++;
continue;
}
***************
*** 2536,2542 ****
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
if (in_quote)
--- 3760,3766 ----
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attr_buf, c);
}
if (in_quote)
***************
*** 2550,2556 ****
else
*isnull = false;
! return attribute_buf.data;
}
/*
--- 3774,3780 ----
else
*isnull = false;
! return attr_buf.data;
}
/*
***************
*** 2578,2606 ****
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! attribute_buf.cursor = 0;
! enlargeStringInfo(&attribute_buf, fld_size);
! CopyGetData(attribute_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attribute_buf.len = fld_size;
! attribute_buf.data[fld_size] = '\0';
/* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attribute_buf),
ObjectIdGetDatum(typioparam));
/* Trouble if it didn't eat the whole buffer */
! if (attribute_buf.cursor != attribute_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
--- 3802,3830 ----
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("invalid field size")));
! /* reset attr_buf to empty, and load raw data in it */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! enlargeStringInfo(&attr_buf, fld_size);
! CopyGetData(attr_buf.data, fld_size);
if (CopyGetEof())
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
errmsg("unexpected EOF in COPY data")));
! attr_buf.len = fld_size;
! attr_buf.data[fld_size] = '\0';
/* Call the column type's binary input converter */
result = FunctionCall2(flinfo,
! PointerGetDatum(&attr_buf),
ObjectIdGetDatum(typioparam));
/* Trouble if it didn't eat the whole buffer */
! if (attr_buf.cursor != attr_buf.len)
ereport(ERROR,
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format")));
***************
*** 2797,2799 ****
--- 4021,4086 ----
return attnums;
}
+
+ /******************************************************
+ * utility functions, should be placed elsewhere in src
+ ******************************************************/
+ /*
+ * These are custom versions of the string function strchr().
+ * As opposed to the original strchr which searches through
+ * a string until the target character is found, or a NULL is
+ * found, this version will not return when a NULL is found.
+ * Instead it will search through a pre-defined length of
+ * bytes and will return only if the target character(s) is reached.
+ *
+ * If our client encoding is not a supported server encoding, we
+ * know that it is not safe to look at each character as trailing
+ * byte in a multibyte character may be a 7-bit ASCII equivalent.
+ * Therefore we use pg_encoding_mblen to skip to the end of the
+ * character.
+ *
+ * parameters:
+ * s - string being searched.
+ * c(n) - char we are searching for.
+ * len - maximum # of bytes to search.
+ *
+ * returns:
+ * pointer to c - if c is located within the string.
+ * NULL - if c was not found in specified length of search. Note:
+ * this DOESN'T mean that a '\0' was reached.
+ */
+ char *
+ strchrlen(const char *s, int c, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, s);
+
+ for (start = s; *s != c && s < start + len; s += mblen)
+ mblen = pg_encoding_mblen(client_encoding, s);
+ }
+ else
+ /* safe to scroll byte by byte */
+ {
+ for (start = s; *s != c && s < start + len; s++)
+ ;
+ }
+
+ return ((*s == c) ? (char *) s : NULL);
+ }
+
+ char *
+ str2chrlen(const char *s, int c1, int c2, size_t len, char *c_found)
+ {
+ const char *start;
+
+ *c_found = '\0';
+
+ for (start = s; *s != c1 && *s != c2 && s < start + len; s++)
+ ;
+
+ *c_found = *s;
+ return (*c_found != '\0' ? (char *) s : NULL);
+ }
pgindent-81.patch (application/octet-stream, 37.1K) download
Index: src/tools/pgindent/pgindent
===================================================================
RCS file: /projects/cvsroot/pgsql/src/tools/pgindent/pgindent,v
retrieving revision 1.73
diff -c -r1.73 pgindent
*** src/tools/pgindent/pgindent 7 Oct 2004 14:15:50 -0000 1.73
--- src/tools/pgindent/pgindent 28 Jun 2005 04:43:48 -0000
***************
*** 56,62 ****
line1 !~ "^typedef" &&
line1 !~ "^extern[ ][ ]*\"C\"" &&
line1 !~ "=" &&
! line1 ~ "\)")
print "int pgindent_func_no_var_fix;";
line1 = line2;
}
--- 56,62 ----
line1 !~ "^typedef" &&
line1 !~ "^extern[ ][ ]*\"C\"" &&
line1 !~ "=" &&
! line1 ~ ")")
print "int pgindent_func_no_var_fix;";
line1 = line2;
}
***************
*** 106,145 ****
# We get the list of typedef's from /src/tools/find_typedef
indent -bad -bap -bc -bl -d0 -cdb -nce -nfc1 -di12 -i4 -l79 \
-lp -nip -npro -bbb $EXTRA_OPTS \
- -TAES_KEY \
- -TAMT \
- -TAMTS \
- -TASN1_BIT_STRING \
- -TASN1_BMPSTRING \
- -TASN1_BOOLEAN \
- -TASN1_CTX \
- -TASN1_ENCODING \
- -TASN1_ENUMERATED \
- -TASN1_GENERALIZEDTIME \
- -TASN1_GENERALSTRING \
- -TASN1_HEADER \
- -TASN1_IA5STRING \
- -TASN1_INTEGER \
- -TASN1_ITEM \
- -TASN1_ITEM_EXP \
- -TASN1_METHOD \
- -TASN1_NULL \
- -TASN1_OBJECT \
- -TASN1_OCTET_STRING \
- -TASN1_PRINTABLESTRING \
- -TASN1_STRING \
- -TASN1_STRING_TABLE \
- -TASN1_T61STRING \
- -TASN1_TEMPLATE \
- -TASN1_TIME \
- -TASN1_TLC \
- -TASN1_TYPE \
- -TASN1_UNIVERSALSTRING \
- -TASN1_UTCTIME \
- -TASN1_UTF8STRING \
- -TASN1_VALUE \
- -TASN1_VISIBLESTRING \
- -TAV \
-TA_Const \
-TA_Expr \
-TA_Expr_Kind \
--- 106,111 ----
***************
*** 153,175 ****
-TAclMode \
-TAclObjectKind \
-TAclResult \
-TAgg \
! -TAggHashEntry \
! -TAggHashEntryData \
-TAggInfo \
-TAggState \
-TAggStatePerAgg \
- -TAggStatePerAggData \
-TAggStatePerGroup \
- -TAggStatePerGroupData \
-TAggStrategy \
-TAggref \
-TAggrefExprState \
-TAlias \
-TAllocBlock \
- -TAllocBlockData \
-TAllocChunk \
- -TAllocChunkData \
-TAllocPointer \
-TAllocSet \
-TAllocSetContext \
--- 119,138 ----
-TAclMode \
-TAclObjectKind \
-TAclResult \
+ -TAfterTriggerEvent \
+ -TAfterTriggerEventList \
-TAgg \
! -TAggClauseCounts \
-TAggInfo \
-TAggState \
-TAggStatePerAgg \
-TAggStatePerGroup \
-TAggStrategy \
-TAggref \
-TAggrefExprState \
-TAlias \
-TAllocBlock \
-TAllocChunk \
-TAllocPointer \
-TAllocSet \
-TAllocSetContext \
***************
*** 177,182 ****
--- 140,146 ----
-TAllocateDescKind \
-TAlterDatabaseSetStmt \
-TAlterDomainStmt \
+ -TAlterFunctionStmt \
-TAlterGroupStmt \
-TAlterOwnerStmt \
-TAlterSeqStmt \
***************
*** 200,229 ****
-TArrayBuildState \
-TArrayExpr \
-TArrayExprState \
-TArrayMetaState \
-TArrayParseState \
-TArrayRef \
-TArrayRefExprState \
-TArrayType \
-TAttInMetadata \
- -TAttrDefInfo \
-TAttrDefault \
-TAttrNumber \
-TAuthRequest \
- -TBF_KEY \
- -TBIGNUM \
- -TBINOP \
- -TBIO \
- -TBIO_F_BUFFER_CTX \
- -TBIO_METHOD \
- -TBIO_dummy \
- -TBIT_STRING_BITNAME \
- -TBLOCK \
-TBMS_Membership \
- -TBN_BLINDING \
- -TBN_CTX \
- -TBN_MONT_CTX \
- -TBN_RECP_CTX \
-TBOX \
-TBTBuildState \
-TBTItem \
--- 164,181 ----
-TArrayBuildState \
-TArrayExpr \
-TArrayExprState \
+ -TArrayMapState \
-TArrayMetaState \
-TArrayParseState \
-TArrayRef \
-TArrayRefExprState \
+ -TArrayTuple \
-TArrayType \
-TAttInMetadata \
-TAttrDefault \
-TAttrNumber \
-TAuthRequest \
-TBMS_Membership \
-TBOX \
-TBTBuildState \
-TBTItem \
***************
*** 238,260 ****
-TBTStack \
-TBTStackData \
-TBTWriteState \
- -TBUF_MEM \
-TBackend \
-TBackendId \
-TBgWriterRequest \
-TBgWriterShmemStruct \
-TBitmapset \
-TBkpBlock \
-TBlock \
- -TBlockId \
-TBlockIdData \
-TBlockNumber \
- -TBlockSampler \
- -TBlockSamplerData \
-TBoolExpr \
-TBoolExprState \
-TBoolExprType \
- -TBoolPtr \
-TBoolTestType \
-TBooleanTest \
-TBpChar \
--- 190,218 ----
-TBTStack \
-TBTStackData \
-TBTWriteState \
-TBackend \
-TBackendId \
-TBgWriterRequest \
-TBgWriterShmemStruct \
+ -TBitmapAnd \
+ -TBitmapAndPath \
+ -TBitmapAndState \
+ -TBitmapHeapPath \
+ -TBitmapHeapScan \
+ -TBitmapHeapScanState \
+ -TBitmapIndexScan \
+ -TBitmapIndexScanState \
+ -TBitmapOr \
+ -TBitmapOrPath \
+ -TBitmapOrState \
-TBitmapset \
-TBkpBlock \
-TBlock \
-TBlockIdData \
-TBlockNumber \
-TBoolExpr \
-TBoolExprState \
-TBoolExprType \
-TBoolTestType \
-TBooleanTest \
-TBpChar \
***************
*** 264,295 ****
-TBuffer \
-TBufferDesc \
-TBufferLookupEnt \
- -TBufferStrategyCDB \
-TBufferStrategyControl \
-TBufferTag \
-TByte \
-TBytef \
-TCAC_state \
- -TCAST_KEY \
-TCATEGORY \
- -TCFuncHashTabEntry \
-TCIRCLE \
- -TCOMP_CTX \
- -TCOMP_METHOD \
- -TCONDOP \
- -TCOP \
- -TCPFunction \
- -TCPPFunction \
- -TCRYPTO_EX_DATA \
- -TCRYPTO_EX_DATA_FUNCS \
- -TCRYPTO_EX_DATA_IMPL \
- -TCRYPTO_EX_dup \
- -TCRYPTO_EX_free \
- -TCRYPTO_EX_new \
- -TCRYPTO_MEM_LEAK_CB \
- -TCRYPTO_dynlock \
- -TCURCUR \
- -TCV \
-TCacheCallbackFunction \
-TCancelRequestPacket \
-TCaseExpr \
--- 222,234 ----
***************
*** 307,316 ****
-TCatalogIndexState \
-TChangeVarNodes_context \
-TCheckPoint \
- -TCheckPointStmt \
-TChromosome \
-TCity \
- -TClientData \
-TClosePortalStmt \
-TClosePtr \
-TClusterStmt \
--- 246,253 ----
***************
*** 320,326 ****
-TCoerceToDomain \
-TCoerceToDomainState \
-TCoerceToDomainValue \
- -TCoercionCodes \
-TCoercionContext \
-TCoercionForm \
-TColumnDef \
--- 257,262 ----
***************
*** 342,349 ****
-TContainsOids \
-TControlFileData \
-TConvInfo \
-TCookedConstraint \
- -TCopyDest \
-TCopyReadResult \
-TCopyStmt \
-TCost \
--- 278,286 ----
-TContainsOids \
-TControlFileData \
-TConvInfo \
+ -TConvertRowtypeExpr \
+ -TConvertRowtypeExprState \
-TCookedConstraint \
-TCopyReadResult \
-TCopyStmt \
-TCost \
***************
*** 357,363 ****
-TCreateOpClassStmt \
-TCreatePLangStmt \
-TCreateSchemaStmt \
- -TCreateSchemaStmtContext \
-TCreateSeqStmt \
-TCreateStmt \
-TCreateStmtContext \
--- 294,299 ----
***************
*** 365,410 ****
-TCreateTrigStmt \
-TCreateUserStmt \
-TCreatedbStmt \
- -TCred \
-TCtlCommand \
-TCustomOutPtr \
-TDBState \
-TDCHCacheEntry \
- -TDCH_poz \
-TDEADLOCK_INFO \
- -TDES_cblock \
- -TDES_key_schedule \
- -TDH \
- -TDH_METHOD \
-TDIR \
-TDR_printtup \
- -TDSA \
- -TDSA_METHOD \
- -TDSA_SIG \
-TDataDumperPtr \
-TDateADT \
-TDatum \
- -TDatumPtr \
-TDatumTuple \
-TDatumTupleFields \
-TDeallocateStmt \
-TDeclareCursorStmt \
-TDefElem \
- -TDeferredTriggerEvent \
- -TDeferredTriggerEventData \
- -TDeferredTriggerEventItem \
- -TDeferredTriggerState \
- -TDeferredTriggerStateData \
- -TDeferredTriggerStatus \
- -TDeferredTriggerStatusData \
- -TDeferredTriggers \
- -TDeferredTriggersData \
-TDefineStmt \
-TDeleteStmt \
-TDependencyType \
-TDestReceiver \
-TDistinctExpr \
- -TDl_info \
-TDlelem \
-TDllist \
-TDomainConstraintState \
--- 301,326 ----
***************
*** 421,452 ****
-TDumpId \
-TDumpableObject \
-TDumpableObjectType \
- -TDupHashTabEntry \
-TDynamicFileList \
-TEDGE \
- -TENGINE \
- -TERR_FNS \
- -TERR_STATE \
- -TERR_STRING_DATA \
-TEState \
- -TEVP_CIPHER \
- -TEVP_CIPHER_CTX \
- -TEVP_CIPHER_INFO \
- -TEVP_ENCODE_CTX \
- -TEVP_MD \
- -TEVP_MD_CTX \
- -TEVP_PBE_KEYGEN \
- -TEVP_PKEY \
-TEdge \
-TEndBlobPtr \
-TEndBlobsPtr \
-TEndDataPtr \
- -TEolType \
-TErrorContextCallback \
-TErrorData \
-TExceptionLabelMap \
- -TExecContext \
- -TExecContextData \
-TExecScanAccessMtd \
-TExecStatus \
-TExecStatusType \
--- 337,352 ----
***************
*** 460,471 ****
-TExprDoneCond \
-TExprState \
-TExprStateEvalFunc \
- -TFF \
-TFILE \
- -TFSMHeader \
-TFSMPageData \
-TFSMRelation \
- -TFUNMAP \
-TFetchDirection \
-TFetchStmt \
-TFieldSelect \
--- 360,368 ----
***************
*** 481,487 ****
-TFormData_pg_aggregate \
-TFormData_pg_am \
-TFormData_pg_amop \
- -TFormData_pg_amproc \
-TFormData_pg_attrdef \
-TFormData_pg_attribute \
-TFormData_pg_cast \
--- 378,383 ----
***************
*** 490,499 ****
-TFormData_pg_conversion \
-TFormData_pg_database \
-TFormData_pg_depend \
- -TFormData_pg_description \
-TFormData_pg_group \
-TFormData_pg_index \
- -TFormData_pg_inherits \
-TFormData_pg_language \
-TFormData_pg_largeobject \
-TFormData_pg_listener \
--- 386,393 ----
***************
*** 511,517 ****
-TForm_pg_aggregate \
-TForm_pg_am \
-TForm_pg_amop \
- -TForm_pg_amproc \
-TForm_pg_attrdef \
-TForm_pg_attribute \
-TForm_pg_cast \
--- 405,410 ----
***************
*** 520,529 ****
-TForm_pg_conversion \
-TForm_pg_database \
-TForm_pg_depend \
- -TForm_pg_description \
-TForm_pg_group \
-TForm_pg_index \
- -TForm_pg_inherits \
-TForm_pg_language \
-TForm_pg_largeobject \
-TForm_pg_listener \
--- 413,420 ----
***************
*** 550,564 ****
-TFuncExprState \
-TFuncInfo \
-TFuncWithArgs \
- -TFunction \
-TFunctionCallInfo \
-TFunctionCallInfoData \
-TFunctionParameter \
-TFunctionScan \
-TFunctionScanState \
- -TGEN_SESSION_CB \
-TGISTBuildState \
-TGISTENTRY \
-TGISTPageOpaque \
-TGISTPageOpaqueData \
-TGISTSTACK \
--- 441,456 ----
-TFuncExprState \
-TFuncInfo \
-TFuncWithArgs \
-TFunctionCallInfo \
-TFunctionCallInfoData \
-TFunctionParameter \
+ -TFunctionParameterMode \
-TFunctionScan \
-TFunctionScanState \
-TGISTBuildState \
-TGISTENTRY \
+ -TGISTInsertStack \
+ -TGISTInsertState \
-TGISTPageOpaque \
-TGISTPageOpaqueData \
-TGISTSTACK \
***************
*** 568,587 ****
-TGISTScanOpaque \
-TGISTScanOpaqueData \
-TGIST_SPLITVEC \
- -TGP \
-TGUC_yy_size_t \
-TGUC_yy_state_type \
- -TGV \
- -TGVOP \
-TGene \
-TGenericExprState \
-TGeqoEvalData \
-TGistEntryVector \
-TGrantObjectType \
-TGrantStmt \
-TGroup \
-TGroupClause \
-TGroupState \
-TGucBoolAssignHook \
-TGucContext \
-TGucIntAssignHook \
--- 460,480 ----
-TGISTScanOpaque \
-TGISTScanOpaqueData \
-TGIST_SPLITVEC \
-TGUC_yy_size_t \
-TGUC_yy_state_type \
-TGene \
-TGenericExprState \
-TGeqoEvalData \
+ -TGistBDItem \
-TGistEntryVector \
+ -TGistVacuum \
+ -TGlobalTransaction \
-TGrantObjectType \
-TGrantStmt \
-TGroup \
-TGroupClause \
-TGroupState \
+ -TGroupVarInfo \
-TGucBoolAssignHook \
-TGucContext \
-TGucIntAssignHook \
***************
*** 597,620 ****
-THASHHDR \
-THASHSEGMENT \
-THASH_SEQ_STATUS \
- -THE \
- -THEK \
- -THISTORY_STATE \
- -THIST_ENTRY \
-THTAB \
-THTSV_Result \
- -THV \
-THash \
-THashAllocFunc \
-THashBuildState \
-THashCompareFunc \
-THashItem \
-THashItemData \
-THashJoin \
-THashJoinState \
-THashJoinTable \
-THashJoinTuple \
- -THashJoinTupleData \
-THashMetaPage \
-THashMetaPageData \
-THashPageOpaque \
--- 490,509 ----
-THASHHDR \
-THASHSEGMENT \
-THASH_SEQ_STATUS \
-THTAB \
+ -THTSU_Result \
-THTSV_Result \
-THash \
-THashAllocFunc \
-THashBuildState \
-THashCompareFunc \
+ -THashCopyFunc \
-THashItem \
-THashItemData \
-THashJoin \
-THashJoinState \
-THashJoinTable \
-THashJoinTuple \
-THashMetaPage \
-THashMetaPageData \
-THashPageOpaque \
***************
*** 625,631 ****
-THashScanOpaque \
-THashScanOpaqueData \
-THashState \
- -THashTableData \
-THashValueFunc \
-THeapScanDesc \
-THeapScanDescData \
--- 514,519 ----
***************
*** 634,652 ****
-THeapTupleFields \
-THeapTupleHeader \
-THeapTupleHeaderData \
- -TI16 \
- -TI32 \
- -TI8 \
- -TIO \
-TIOFuncSelector \
- -TIV \
- -TIV64 \
-TIdList \
-TInClauseInfo \
-TIncrementVarSublevelsUp_context \
-TIndex \
- -TIndexAttributeBitMap \
- -TIndexAttributeBitMapData \
-TIndexBuildCallback \
-TIndexBulkDeleteCallback \
-TIndexBulkDeleteResult \
--- 522,532 ----
***************
*** 659,665 ****
-TIndexScan \
-TIndexScanDesc \
-TIndexScanDescData \
- -TIndexScanDescPtr \
-TIndexScanState \
-TIndexStmt \
-TIndexTuple \
--- 539,544 ----
***************
*** 669,679 ****
-TIndxInfo \
-TInhInfo \
-TInhOption \
- -TInhPaths \
-TInhRelation \
-TInnerIndexscanInfo \
- -TInsertIndexResult \
- -TInsertIndexResultData \
-TInsertStmt \
-TInstrumentation \
-TInt8TransTypeData \
--- 548,555 ----
***************
*** 686,722 ****
-TInvalidationListHeader \
-TIpcMemoryId \
-TIpcMemoryKey \
- -TIpcSemaphoreId \
- -TIpcSemaphoreKey \
-TItem \
-TItemId \
-TItemIdData \
-TItemIdFlags \
- -TItemLength \
- -TItemOffset \
-TItemPointer \
-TItemPointerData \
- -TJMPENV \
-TJoin \
-TJoinExpr \
- -TJoinInfo \
-TJoinPath \
-TJoinState \
-TJoinType \
-TJunkFilter \
- -TKEYMAP_ENTRY \
- -TKEYMAP_ENTRY_ARRAY \
-TKeySuffix \
-TKeyWord \
- -TKeymap \
- -TLHASH \
- -TLHASH_COMP_FN_TYPE \
- -TLHASH_DOALL_ARG_FN_TYPE \
- -TLHASH_DOALL_FN_TYPE \
- -TLHASH_HASH_FN_TYPE \
- -TLHASH_NODE \
-TLINE \
- -TLISTOP \
-TLOCALLOCK \
-TLOCALLOCKOWNER \
-TLOCALLOCKTAG \
--- 562,582 ----
***************
*** 725,732 ****
-TLOCKMETHODID \
-TLOCKMODE \
-TLOCKTAG \
- -TLOGOP \
- -TLOOP \
-TLSEG \
-TLVRelStats \
-TLWLock \
--- 585,590 ----
***************
*** 741,762 ****
-TListenStmt \
-TLoadStmt \
-TLocationIndex \
-TLockData \
- -TLockInfo \
-TLockInfoData \
-TLockMethod \
-TLockMethodData \
-TLockRelId \
-TLockStmt \
-TLogStmtLevel \
-TLogicalTape \
-TLogicalTapeSet \
- -TMAGIC \
- -TMD2_CTX \
- -TMD4_CTX \
- -TMD5_CTX \
- -TMDC2_CTX \
- -TMGVTBL \
-TMaterial \
-TMaterialPath \
-TMaterialState \
--- 599,615 ----
-TListenStmt \
-TLoadStmt \
-TLocationIndex \
+ -TLockAcquireResult \
-TLockData \
-TLockInfoData \
-TLockMethod \
-TLockMethodData \
-TLockRelId \
-TLockStmt \
+ -TLockTupleMode \
-TLogStmtLevel \
-TLogicalTape \
-TLogicalTapeSet \
-TMaterial \
-TMaterialPath \
-TMaterialState \
***************
*** 764,782 ****
-TMemoryContext \
-TMemoryContextData \
-TMemoryContextMethods \
-TMergeJoin \
-TMergeJoinState \
-TMergePath \
-TMsgType \
! -TMyVarInfo \
! -TNETSCAPE_CERT_SEQUENCE \
! -TNETSCAPE_SPKAC \
! -TNETSCAPE_SPKI \
-TNUMCacheEntry \
-TNUMDesc \
-TNUMProc \
- -TNUM_poz \
- -TNV \
-TName \
-TNameData \
-TNamespaceInfo \
--- 617,638 ----
-TMemoryContext \
-TMemoryContextData \
-TMemoryContextMethods \
+ -TMergeFunctionKind \
-TMergeJoin \
+ -TMergeJoinClause \
-TMergeJoinState \
-TMergePath \
+ -TMinMaxAggInfo \
-TMsgType \
! -TMultiXactId \
! -TMultiXactOffset \
! -TNSLinkEditErrors \
! -TNSObjectFileImage \
! -TNSObjectFileImageReturnCode \
! -TNSSymbol \
-TNUMCacheEntry \
-TNUMDesc \
-TNUMProc \
-TName \
-TNameData \
-TNamespaceInfo \
***************
*** 785,790 ****
--- 641,647 ----
-TNestPath \
-TNewColumnValue \
-TNewConstraint \
+ -TNewPage \
-TNode \
-TNodeTag \
-TNotifyStmt \
***************
*** 795,802 ****
-TNumericData \
-TNumericDigit \
-TNumericVar \
- -TOBJ_NAME \
- -TOP \
-TObjectAddress \
-TObjectAddresses \
-TObjectClass \
--- 652,657 ----
***************
*** 809,815 ****
-TOldstyle_fnextra \
-TOnCommitAction \
-TOnCommitItem \
- -TOpClassCacheEnt \
-TOpClassMember \
-TOpExpr \
-TOpclassCandidateList \
--- 664,669 ----
***************
*** 817,833 ****
-TOperator \
-TOprInfo \
-TOutputContext \
- -TOutrec \
- -TPADOFFSET \
-TPATH \
- -TPBE2PARAM \
- -TPBEPARAM \
- -TPBKDF2PARAM \
- -TPEM_CTX \
- -TPEM_ENCODE_SEAL_CTX \
- -TPEM_USER \
- -TPERL_CONTEXT \
- -TPERL_SI \
-TPGAsyncStatusType \
-TPGErrorVerbosity \
-TPGFInfoFunction \
--- 671,677 ----
***************
*** 839,844 ****
--- 683,689 ----
-TPGMessageField \
-TPGNoticeHooks \
-TPGPROC \
+ -TPGQueryClass \
-TPGSemaphore \
-TPGSemaphoreData \
-TPGSetenvStatusType \
***************
*** 846,851 ****
--- 691,697 ----
-TPGTransactionStatusType \
-TPGVerbosity \
-TPG_Lock_Status \
+ -TPGcancel \
-TPGconn \
-TPGlobjfuncs \
-TPGnotify \
***************
*** 853,877 ****
-TPGresAttValue \
-TPGresult \
-TPGresult_data \
- -TPKCS7 \
- -TPKCS7_DIGEST \
- -TPKCS7_ENCRYPT \
- -TPKCS7_ENC_CONTENT \
- -TPKCS7_ENVELOPE \
- -TPKCS7_ISSUER_AND_SERIAL \
- -TPKCS7_RECIP_INFO \
- -TPKCS7_SIGNED \
- -TPKCS7_SIGNER_INFO \
- -TPKCS7_SIGN_ENVELOPE \
- -TPKCS8_PRIV_KEY_INFO \
-TPLPGSQL_YYSTYPE \
- -TPLpgSQL_arrayelem \
-TPLpgSQL_condition \
-TPLpgSQL_datum \
-TPLpgSQL_diag_item \
-TPLpgSQL_dstring \
-TPLpgSQL_exception \
! -TPLpgSQL_exceptions \
-TPLpgSQL_execstate \
-TPLpgSQL_expr \
-TPLpgSQL_func_hashkey \
--- 699,711 ----
-TPGresAttValue \
-TPGresult \
-TPGresult_data \
-TPLPGSQL_YYSTYPE \
-TPLpgSQL_condition \
-TPLpgSQL_datum \
-TPLpgSQL_diag_item \
-TPLpgSQL_dstring \
-TPLpgSQL_exception \
! -TPLpgSQL_exception_block \
-TPLpgSQL_execstate \
-TPLpgSQL_expr \
-TPLpgSQL_func_hashkey \
***************
*** 879,913 ****
-TPLpgSQL_ns \
-TPLpgSQL_nsitem \
-TPLpgSQL_rec \
- -TPLpgSQL_recfield \
-TPLpgSQL_row \
-TPLpgSQL_stmt \
- -TPLpgSQL_stmt_assign \
-TPLpgSQL_stmt_block \
- -TPLpgSQL_stmt_close \
-TPLpgSQL_stmt_dynexecute \
-TPLpgSQL_stmt_dynfors \
- -TPLpgSQL_stmt_execsql \
- -TPLpgSQL_stmt_exit \
- -TPLpgSQL_stmt_fetch \
- -TPLpgSQL_stmt_fori \
-TPLpgSQL_stmt_fors \
- -TPLpgSQL_stmt_getdiag \
- -TPLpgSQL_stmt_if \
- -TPLpgSQL_stmt_loop \
-TPLpgSQL_stmt_open \
- -TPLpgSQL_stmt_perform \
-TPLpgSQL_stmt_raise \
-TPLpgSQL_stmt_return \
-TPLpgSQL_stmt_return_next \
-TPLpgSQL_stmt_select \
- -TPLpgSQL_stmt_while \
- -TPLpgSQL_stmts \
-TPLpgSQL_trigarg \
-TPLpgSQL_type \
-TPLpgSQL_var \
-TPLpgSQL_variable \
- -TPMOP \
-TPMSignalReason \
-TPOLYGON \
-TPQArgBlock \
--- 713,733 ----
***************
*** 922,934 ****
-TPROCLOCKTAG \
-TPROC_HDR \
-TPROC_QUEUE \
- -TPV \
- -TPVOP \
- -TPacketLen \
-TPage \
-TPageFreeSpaceInfo \
-TPageHeader \
-TPageHeaderData \
-TParam \
-TParamExecData \
-TParamListInfo \
--- 742,753 ----
-TPROCLOCKTAG \
-TPROC_HDR \
-TPROC_QUEUE \
-TPage \
-TPageFreeSpaceInfo \
-TPageHeader \
-TPageHeaderData \
+ -TPageSplitRecord \
+ -TPagetableEntry \
-TParam \
-TParamExecData \
-TParamListInfo \
***************
*** 942,962 ****
-TPattern_Type \
-TPendingOperationEntry \
-TPendingRelDelete \
- -TPerlExitListEntry \
- -TPerlInterpreter \
-TPgStat_Counter \
-TPgStat_Info \
- -TPgStat_Msg \
-TPgStat_MsgActivity \
-TPgStat_MsgBestart \
-TPgStat_MsgBeterm \
- -TPgStat_MsgDropdb \
-TPgStat_MsgDummy \
-TPgStat_MsgHdr \
-TPgStat_MsgResetcounter \
-TPgStat_MsgTabpurge \
-TPgStat_MsgTabstat \
- -TPgStat_StatBeDead \
-TPgStat_StatBeEntry \
-TPgStat_StatDBEntry \
-TPgStat_StatTabEntry \
--- 761,776 ----
***************
*** 964,976 ****
-TPg_finfo_record \
-TPlan \
-TPlanState \
-TPlannerParamItem \
-TPoint \
-TPointer \
-TPool \
-TPort \
-TPortal \
- -TPortalData \
-TPortalHashEnt \
-TPortalStatus \
-TPortalStrategy \
--- 778,790 ----
-TPg_finfo_record \
-TPlan \
-TPlanState \
+ -TPlannerInfo \
-TPlannerParamItem \
-TPoint \
-TPointer \
-TPool \
-TPort \
-TPortal \
-TPortalHashEnt \
-TPortalStatus \
-TPortalStrategy \
***************
*** 982,987 ****
--- 796,802 ----
-TPrinttupAttrInfo \
-TPrivGrantee \
-TPrivTarget \
+ -TProcArrayStruct \
-TProcLangInfo \
-TProcState \
-TProcessingMode \
***************
*** 989,1008 ****
-TProtocolVersion \
-TPsqlScanResult \
-TPsqlScanState \
- -TPsqlScanStateData \
- -TPsqlSettings \
-TQualCost \
-TQuery \
-TQueryDesc \
-TQuerySource \
- -TRC2_KEY \
- -TRC4_KEY \
- -TREGEXP \
- -TRIPEMD160_CTX \
-TRI_QueryHashEntry \
-TRI_QueryKey \
- -TRSA \
- -TRSA_METHOD \
-TRTBuildState \
-TRTEKind \
-TRTSTACK \
--- 804,815 ----
***************
*** 1029,1042 ****
-TReindexStmt \
-TRelFileNode \
-TRelIdCacheEnt \
- -TRelNameCacheEnt \
-TRelOptInfo \
-TRelOptKind \
-TRelToCheck \
-TRelToCluster \
-TRelabelType \
-TRelation \
! -TRelationBuildDescInfo \
-TRelationData \
-TRelationPtr \
-TRelativeTime \
--- 836,848 ----
-TReindexStmt \
-TRelFileNode \
-TRelIdCacheEnt \
-TRelOptInfo \
-TRelOptKind \
-TRelToCheck \
-TRelToCluster \
-TRelabelType \
-TRelation \
! -TRelationAmInfo \
-TRelationData \
-TRelationPtr \
-TRelativeTime \
***************
*** 1047,1056 ****
-TRemoveOperStmt \
-TRenameStmt \
-TResTarget \
- -TResdom \
-TResolveNew_context \
-TResourceOwner \
- -TResourceOwnerData \
-TResourceReleaseCallback \
-TResourceReleaseCallbackItem \
-TResourceReleasePhase \
--- 853,860 ----
***************
*** 1069,1101 ****
-TRuleInfo \
-TRuleLock \
-TRuleStmt \
- -TSHA_CTX \
-TSHMEM_OFFSET \
-TSHM_QUEUE \
-TSISeg \
-TSMgrRelation \
-TSMgrRelationData \
-TSPITupleTable \
- -TSPLITCOST \
-TSPLITVEC \
-TSQLFunctionCache \
-TSQLFunctionCachePtr \
- -TSSL \
- -TSSL2_STATE \
- -TSSL3_BUFFER \
- -TSSL3_RECORD \
- -TSSL3_STATE \
- -TSSL_CIPHER \
- -TSSL_COMP \
- -TSSL_CTX \
- -TSSL_METHOD \
- -TSSL_SESSION \
- -TSTACK \
- -TSTRLEN \
- -TSUBLEXINFO \
- -TSV \
- -TSVOP \
- -TSaveArchivePtr \
-TScalarArrayOpExpr \
-TScalarArrayOpExprState \
-TScalarItem \
--- 873,887 ----
***************
*** 1113,1118 ****
--- 899,907 ----
-TSeqScanState \
-TSeqTable \
-TSeqTableData \
+ -TSetConstraintState \
+ -TSetConstraintStateData \
+ -TSetConstraintTriggerData \
-TSetFunctionReturnMode \
-TSetOp \
-TSetOpCmd \
***************
*** 1122,1137 ****
-TSetToDefault \
-TSharedInvalCatcacheMsg \
-TSharedInvalRelcacheMsg \
-TSharedInvalidationMessage \
-TShmemIndexEnt \
-TShutdownMode \
- -TSigHandler \
- -TSighandler_t \
- -TSigsave_t \
-TSize \
-TSlruCtl \
-TSlruCtlData \
- -TSlruErrorCause \
-TSlruFlush \
-TSlruFlushData \
-TSlruPageStatus \
--- 911,923 ----
-TSetToDefault \
-TSharedInvalCatcacheMsg \
-TSharedInvalRelcacheMsg \
+ -TSharedInvalSmgrMsg \
-TSharedInvalidationMessage \
-TShmemIndexEnt \
-TShutdownMode \
-TSize \
-TSlruCtl \
-TSlruCtlData \
-TSlruFlush \
-TSlruFlushData \
-TSlruPageStatus \
***************
*** 1145,1156 ****
-TSortClause \
-TSortFunctionKind \
-TSortState \
-TStackElem \
-TStandardChunkHeader \
-TStartBlobPtr \
-TStartBlobsPtr \
-TStartDataPtr \
- -TStartupPacket \
-TStdAnalyzeData \
-TStrategyNumber \
-TStringInfo \
--- 931,942 ----
-TSortClause \
-TSortFunctionKind \
-TSortState \
+ -TSplitedPageLayout \
-TStackElem \
-TStandardChunkHeader \
-TStartBlobPtr \
-TStartBlobsPtr \
-TStartDataPtr \
-TStdAnalyzeData \
-TStrategyNumber \
-TStringInfo \
***************
*** 1159,1269 ****
-TSubLinkType \
-TSubPlan \
-TSubPlanState \
-TSubqueryScan \
-TSubqueryScanState \
-TSysScanDesc \
-TSysScanDescData \
-TTAR_MEMBER \
-TTBlockState \
! -TTStoreState \
! -TTableDataInfo \
-TTableInfo \
-TTargetEntry \
- -TTclStubHooks \
- -TTclStubs \
- -TTcl_AppInitProc \
- -TTcl_AsyncHandler \
- -TTcl_AsyncProc \
- -TTcl_CallFrame \
- -TTcl_Channel \
- -TTcl_ChannelProc \
- -TTcl_ChannelType \
- -TTcl_ChannelTypeVersion \
- -TTcl_CloseProc \
- -TTcl_CmdDeleteProc \
- -TTcl_CmdInfo \
- -TTcl_CmdProc \
- -TTcl_CmdTraceProc \
- -TTcl_Command \
- -TTcl_Condition \
- -TTcl_CreateFileHandlerProc \
- -TTcl_DString \
- -TTcl_DeleteFileHandlerProc \
- -TTcl_DriverBlockModeProc \
- -TTcl_DriverClose2Proc \
- -TTcl_DriverCloseProc \
- -TTcl_DriverFlushProc \
- -TTcl_DriverGetHandleProc \
- -TTcl_DriverGetOptionProc \
- -TTcl_DriverHandlerProc \
- -TTcl_DriverInputProc \
- -TTcl_DriverOutputProc \
- -TTcl_DriverSeekProc \
- -TTcl_DriverSetOptionProc \
- -TTcl_DriverWatchProc \
- -TTcl_DupInternalRepProc \
- -TTcl_Encoding \
- -TTcl_EncodingConvertProc \
- -TTcl_EncodingFreeProc \
- -TTcl_EncodingState \
- -TTcl_EncodingType \
- -TTcl_EolTranslation \
- -TTcl_Event \
- -TTcl_EventCheckProc \
- -TTcl_EventDeleteProc \
- -TTcl_EventProc \
- -TTcl_EventSetupProc \
- -TTcl_ExitProc \
- -TTcl_FileFreeProc \
- -TTcl_FileProc \
- -TTcl_FreeInternalRepProc \
- -TTcl_FreeProc \
- -TTcl_HashEntry \
- -TTcl_HashSearch \
- -TTcl_HashTable \
- -TTcl_IdleProc \
- -TTcl_Interp \
- -TTcl_InterpDeleteProc \
- -TTcl_MainLoopProc \
- -TTcl_MathProc \
- -TTcl_Mutex \
- -TTcl_Namespace \
- -TTcl_NamespaceDeleteProc \
- -TTcl_NotifierProcs \
- -TTcl_Obj \
- -TTcl_ObjCmdProc \
- -TTcl_ObjType \
- -TTcl_PackageInitProc \
- -TTcl_PanicProc \
- -TTcl_Parse \
- -TTcl_PathType \
- -TTcl_Pid \
- -TTcl_QueuePosition \
- -TTcl_RegExp \
- -TTcl_RegExpIndices \
- -TTcl_RegExpInfo \
- -TTcl_SavedResult \
- -TTcl_SetFromAnyProc \
- -TTcl_SetTimerProc \
- -TTcl_Stat_ \
- -TTcl_TcpAcceptProc \
- -TTcl_ThreadCreateProc \
- -TTcl_ThreadDataKey \
- -TTcl_ThreadId \
- -TTcl_Time \
- -TTcl_TimerProc \
- -TTcl_TimerToken \
- -TTcl_Token \
- -TTcl_Trace \
- -TTcl_UniChar \
- -TTcl_UpdateStringProc \
- -TTcl_Value \
- -TTcl_ValueType \
- -TTcl_Var \
- -TTcl_VarTraceProc \
- -TTcl_WaitForEventProc \
-TThingFile \
- -TThread \
-TTidPath \
-TTidScan \
-TTidScanState \
--- 945,966 ----
-TSubLinkType \
-TSubPlan \
-TSubPlanState \
+ -TSubTransactionId \
+ -TSubXactCallback \
+ -TSubXactCallbackItem \
+ -TSubXactEvent \
-TSubqueryScan \
-TSubqueryScanState \
-TSysScanDesc \
-TSysScanDescData \
-TTAR_MEMBER \
+ -TTBMIterateResult \
+ -TTBMStatus \
-TTBlockState \
! -TTIDBitmap \
-TTableInfo \
-TTargetEntry \
-TThingFile \
-TTidPath \
-TTidScan \
-TTidScanState \
***************
*** 1290,1296 ****
-TTriggerData \
-TTriggerDesc \
-TTriggerEvent \
- -TTriggerInfo \
-TTruncateStmt \
-TTupOutputState \
-TTupSortStatus \
--- 987,992 ----
***************
*** 1301,1326 ****
-TTupleHashEntryData \
-TTupleHashIterator \
-TTupleHashTable \
- -TTupleHashTableData \
-TTupleTable \
-TTupleTableData \
-TTupleTableSlot \
-TTuplesortstate \
-TTuplestorestate \
-TType \
-TTypeCacheEntry \
-TTypeCast \
-TTypeInfo \
-TTypeName \
- -TU16 \
- -TU32 \
- -TU8 \
- -TUI \
- -TUI_METHOD \
- -TUI_STRING \
- -TUNDO_LIST \
- -TUNOP \
- -TUV \
-TUnique \
-TUniquePath \
-TUniquePathMethod \
--- 997,1018 ----
-TTupleHashEntryData \
-TTupleHashIterator \
-TTupleHashTable \
-TTupleTable \
-TTupleTableData \
-TTupleTableSlot \
-TTuplesortstate \
-TTuplestorestate \
+ -TTwoPhaseCallback \
+ -TTwoPhaseFileHeader \
+ -TTwoPhaseLockRecord \
+ -TTwoPhaseRecordOnDisk \
+ -TTwoPhaseRmgrId \
-TType \
-TTypeCacheEntry \
-TTypeCast \
+ -TTypeFuncClass \
-TTypeInfo \
-TTypeName \
-TUnique \
-TUniquePath \
-TUniquePathMethod \
***************
*** 1328,1334 ****
-TUnlistenStmt \
-TUpdateStmt \
-TUserAuth \
- -TVFunction \
-TVRelStats \
-TVTupleLink \
-TVTupleLinkData \
--- 1020,1025 ----
***************
*** 1355,1421 ****
-TVariableStatData \
-TVfd \
-TViewStmt \
! -TWAIT_ORDER \
-TWriteBufPtr \
-TWriteBytePtr \
-TWriteDataPtr \
-TWriteExtraTocPtr \
- -TX509 \
- -TX509_ALGOR \
- -TX509_ATTRIBUTE \
- -TX509_CERT_AUX \
- -TX509_CERT_FILE_CTX \
- -TX509_CINF \
- -TX509_CRL \
- -TX509_CRL_INFO \
- -TX509_EXTENSION \
- -TX509_HASH_DIR_CTX \
- -TX509_INFO \
- -TX509_LOOKUP \
- -TX509_LOOKUP_METHOD \
- -TX509_NAME \
- -TX509_NAME_ENTRY \
- -TX509_OBJECT \
- -TX509_OBJECTS \
- -TX509_PKEY \
- -TX509_PUBKEY \
- -TX509_REQ \
- -TX509_REQ_INFO \
- -TX509_REVOKED \
- -TX509_SIG \
- -TX509_STORE \
- -TX509_STORE_CTX \
- -TX509_TRUST \
- -TX509_VAL \
-TXLogContRecord \
-TXLogCtlData \
-TXLogCtlInsert \
-TXLogCtlWrite \
- -TXLogLongPageHeader \
- -TXLogLongPageHeaderData \
-TXLogPageHeader \
-TXLogPageHeaderData \
-TXLogRecData \
-TXLogRecPtr \
-TXLogRecord \
- -TXLogRelCacheEntry \
-TXLogRelDesc \
-TXLogwrtResult \
-TXLogwrtRqst \
- -TXPV \
- -TXPVAV \
- -TXPVBM \
- -TXPVCV \
- -TXPVFM \
- -TXPVGV \
- -TXPVHV \
- -TXPVIO \
- -TXPVIV \
- -TXPVLV \
- -TXPVMG \
- -TXPVNV \
- -TXPVUV \
- -TXRV \
-TXactCallback \
-TXactCallbackItem \
-TXactEvent \
--- 1046,1068 ----
-TVariableStatData \
-TVfd \
-TViewStmt \
! -TWorking_State \
-TWriteBufPtr \
-TWriteBytePtr \
-TWriteDataPtr \
-TWriteExtraTocPtr \
-TXLogContRecord \
-TXLogCtlData \
-TXLogCtlInsert \
-TXLogCtlWrite \
-TXLogPageHeader \
-TXLogPageHeaderData \
-TXLogRecData \
-TXLogRecPtr \
-TXLogRecord \
-TXLogRelDesc \
-TXLogwrtResult \
-TXLogwrtRqst \
-TXactCallback \
-TXactCallbackItem \
-TXactEvent \
***************
*** 1423,1548 ****
-TYYSTYPE \
-TYY_BUFFER_STATE \
-TYY_CHAR \
- -T_LIB_VERSION_TYPE \
- -T_RuneEntry \
- -T_RuneLocale \
- -T_RuneRange \
-T_SPI_connection \
-T_SPI_plan \
! -T_ossl_old_des_cblock \
! -T_ossl_old_des_key_schedule \
-Taclitem \
-Tadjust_inherited_attrs_context \
-Talc_extra \
-Talloc_func \
- -Tam_extra \
-Tanyarray \
- -Tassoc_list \
-Tatc_extra \
-Tattribute_used_context \
-TbackslashResult \
- -Tbio_info_cb \
-Tbitmapword \
-Tbits16 \
- -Tbits32 \
-Tbits8 \
-Tbool \
-Tbt_incomplete_split \
-Tbytea \
-Tcached_re_str \
- -Tcaddr_t \
-Tcc_t \
-Tcelt \
-Tchar \
- -Tcharf \
-TcheckExprHasAggs_context \
-Tcheck_parameter_resolution_context \
-Tcheck_ungrouped_columns_context \
-Tchr \
- -Tclock_t \
- -Tclockid_t \
-Tcodes_t \
-Tcolor \
- -Tcomment_t \
- -Tconst_DES_cblock \
-Tcontain_var_reference_context \
- -Tcrc64 \
- -Tcv_flags_t \
- -Tdaddr_t \
-Tdate \
-Tdatetkn \
-Tdecimal \
-Tdeparse_context \
-Tdeparse_namespace \
-Tdev_t \
- -Tdiv_t \
-Tdouble \
-TevalPlanQual \
-Teval_const_expressions_context \
- -TexecRowMark \
-Texecution_state \
- -Texpectation \
-Tf_smgr \
- -Tfd_mask \
-Tfd_set \
- -Tfilter_t \
-Tfinalize_primnode_context \
-Tfind_expr_references_context \
-Tfind_minimum_var_level_context \
- -Tfixpt_t \
-Tflatten_join_alias_vars_context \
-Tfloat \
- -Tfloat32 \
- -Tfloat32data \
-Tfloat4 \
- -Tfloat64 \
- -Tfloat64data \
-Tfloat8 \
! -Tfp_except \
! -Tfp_rnd \
! -Tfpclass_t \
-Tfpos_t \
-Tfree_func \
-Tfsec_t \
-Tfunc_ptr \
-Tgenerate_series_fctx \
-Tgid_t \
-TgzFile \
-Thashnode \
-ThbaPort \
-Thelp_handler \
-Tin_addr_t \
-Tin_port_t \
-Tinet \
- -Tinet_struct \
-Tino_t \
-Tint \
-Tint16 \
- -Tint16_t \
- -Tint16m_t \
-Tint2 \
-Tint2vector \
-Tint32 \
- -Tint32_t \
- -Tint32m_t \
-Tint4 \
-Tint64 \
- -Tint64_t \
- -Tint64m_t \
-Tint8 \
- -Tint8_t \
- -Tint8m_t \
-Tinterval \
- -Tintf \
-TitemIdSort \
-TitemIdSortData \
- -Tjmp_buf \
-Tjoin_references_context \
-Tkey_t \
-TlclContext \
-TlclTocEntry \
! -Tldiv_t \
! -Tline_t \
-Tmacaddr \
-Tmb2wchar_with_len_converter \
-Tmbdisplaylen_converter \
--- 1070,1185 ----
-TYYSTYPE \
-TYY_BUFFER_STATE \
-TYY_CHAR \
-T_SPI_connection \
-T_SPI_plan \
! -T__builtin_va_list \
! -T__darwin_blkcnt_t \
! -T__darwin_blksize_t \
! -T__darwin_dev_t \
! -T__darwin_gid_t \
! -T__darwin_ino_t \
! -T__darwin_mode_t \
! -T__darwin_off_t \
! -T__darwin_pid_t \
! -T__darwin_pthread_mutex_t \
! -T__darwin_sigset_t \
! -T__darwin_size_t \
! -T__darwin_socklen_t \
! -T__darwin_ssize_t \
! -T__darwin_suseconds_t \
! -T__darwin_time_t \
! -T__darwin_uid_t \
! -T__darwin_va_list \
! -T__darwin_wchar_t \
! -T__int32_t \
! -T__int64_t \
! -T__uint16_t \
! -T__uint32_t \
! -T__uint8_t \
-Taclitem \
-Tadjust_inherited_attrs_context \
-Talc_extra \
-Talloc_func \
-Tanyarray \
-Tatc_extra \
-Tattribute_used_context \
-TbackslashResult \
-Tbitmapword \
-Tbits16 \
-Tbits8 \
+ -Tblkcnt_t \
+ -Tblksize_t \
-Tbool \
-Tbt_incomplete_split \
-Tbytea \
-Tcached_re_str \
-Tcc_t \
-Tcelt \
-Tchar \
-TcheckExprHasAggs_context \
-Tcheck_parameter_resolution_context \
-Tcheck_ungrouped_columns_context \
-Tchr \
-Tcodes_t \
-Tcolor \
-Tcontain_var_reference_context \
-Tdate \
-Tdatetkn \
-Tdecimal \
-Tdeparse_context \
-Tdeparse_namespace \
-Tdev_t \
-Tdouble \
-TevalPlanQual \
-Teval_const_expressions_context \
-Texecution_state \
-Tf_smgr \
-Tfd_set \
-Tfinalize_primnode_context \
-Tfind_expr_references_context \
-Tfind_minimum_var_level_context \
-Tflatten_join_alias_vars_context \
-Tfloat \
-Tfloat4 \
-Tfloat8 \
! -TfmNodePtr \
-Tfpos_t \
-Tfree_func \
-Tfsec_t \
-Tfunc_ptr \
-Tgenerate_series_fctx \
-Tgid_t \
+ -TgistIncompleteInsert \
+ -TgistxlogEntryUpdate \
+ -TgistxlogInsertComplete \
+ -TgistxlogPage \
+ -TgistxlogPageSplit \
-TgzFile \
-Thashnode \
-ThbaPort \
-Thelp_handler \
-Tin_addr_t \
-Tin_port_t \
+ -Tindexed_tlist \
-Tinet \
-Tino_t \
+ -Tinstr_time \
-Tint \
-Tint16 \
-Tint2 \
-Tint2vector \
-Tint32 \
-Tint4 \
-Tint64 \
-Tint8 \
-Tinterval \
-TitemIdSort \
-TitemIdSortData \
-Tjoin_references_context \
-Tkey_t \
-TlclContext \
-TlclTocEntry \
! -TmXactCacheEnt \
-Tmacaddr \
-Tmb2wchar_with_len_converter \
-Tmbdisplaylen_converter \
***************
*** 1550,1572 ****
-TmixedStruct \
-Tmode_t \
-Tnlink_t \
- -Tnls_uint32 \
-Tnumeric \
-Toff_t \
-Toidvector \
- -Top_tr_array \
- -Topcode \
- -Topindex \
- -ToptType \
-Tpcolor \
- -Tpem_password_cb \
-TpgParameterStatus \
-Tpg_conv_map \
-Tpg_enc \
-Tpg_enc2name \
-Tpg_encname \
-Tpg_local_to_utf \
-Tpg_time_t \
-Tpg_utf_to_local \
-Tpg_wchar \
-Tpg_wchar_tbl \
--- 1187,1205 ----
-TmixedStruct \
-Tmode_t \
-Tnlink_t \
-Tnumeric \
-Toff_t \
-Toidvector \
-Tpcolor \
-TpgParameterStatus \
-Tpg_conv_map \
+ -Tpg_crc32 \
-Tpg_enc \
-Tpg_enc2name \
-Tpg_encname \
-Tpg_local_to_utf \
-Tpg_time_t \
+ -Tpg_tz \
-Tpg_utf_to_local \
-Tpg_wchar \
-Tpg_wchar_tbl \
***************
*** 1574,1628 ****
-Tpgsql_thing_t \
-Tpgthreadlock_t \
-Tpid_t \
- -Tplperl_proc_desc \
- -Tplpgsql_HashEnt \
- -Tplpgsql_yysigned_char \
- -Tpltcl_proc_desc \
- -Tpltcl_query_desc \
-Tpqbool \
-Tpqsigfunc \
-TprintQueryOpt \
-TprintTableOpt \
-TpromptStatus_t \
- -Tpthread_attr_t \
- -Tpthread_cond_t \
- -Tpthread_condattr_t \
- -Tpthread_key_t \
- -Tpthread_mutex_t \
- -Tpthread_mutexattr_t \
- -Tpthread_once_t \
- -Tpthread_t \
- -Tptrdiff_t \
-Tpull_var_clause_context \
-Tpull_varnos_context \
- -Tpvcontents \
- -Tq128_t \
- -Tqaddr_t \
- -Tquad_t \
-TrangeTableEntry_used_context \
-Treduce_outer_joins_state \
- -Tregcomp_t \
-Tregex_t \
- -Tregexec_t \
- -Tregexp \
- -Tregister_t \
-Tregmatch_t \
- -Tregnode \
-Tregoff_t \
-Tregproc \
-Treplace_vars_with_subplan_refs_context \
- -Trewrite_event \
-Trm_detail_t \
- -Trune_t \
- -Trunops_proc_t \
-Tsa_family_t \
! -Tsegsz_t \
! -Tsequence_magic \
-Tsig_atomic_t \
- -Tsig_t \
- -Tsighandler_cxt \
-Tsigjmp_buf \
- -Tsignedbitmapword \
-Tsigset_t \
-Tsize_t \
-Tslock_t \
--- 1207,1231 ----
-Tpgsql_thing_t \
-Tpgthreadlock_t \
-Tpid_t \
-Tpqbool \
-Tpqsigfunc \
-TprintQueryOpt \
-TprintTableOpt \
-TpromptStatus_t \
-Tpull_var_clause_context \
-Tpull_varnos_context \
-TrangeTableEntry_used_context \
-Treduce_outer_joins_state \
-Tregex_t \
-Tregmatch_t \
-Tregoff_t \
-Tregproc \
-Treplace_vars_with_subplan_refs_context \
-Trm_detail_t \
-Tsa_family_t \
! -Tsem_t \
-Tsig_atomic_t \
-Tsigjmp_buf \
-Tsigset_t \
-Tsize_t \
-Tslock_t \
***************
*** 1632,1687 ****
-TsqlparseInfo \
-TsqlparseState \
-Tssize_t \
- -Tssl_crock_st \
- -Tstack_t \
- -Tstrconst \
-Tsubstitute_actual_parameters_context \
- -Tsvindex \
- -Tsvtype \
- -Tswblk_t \
-Ttcflag_t \
- -Ttcp_seq \
-TteReqs \
-Ttext \
-Ttime_t \
-Ttimestamp \
-Tts_db_fctx \
-TuInt \
- -TuIntf \
-TuLong \
- -TuLongf \
-Tu_char \
-Tu_int \
- -Tu_int16_t \
- -Tu_int16m_t \
- -Tu_int32_t \
- -Tu_int32m_t \
- -Tu_int64_t \
- -Tu_int64m_t \
- -Tu_int8_t \
- -Tu_int8m_t \
- -Tu_long \
- -Tu_quad_t \
- -Tu_short \
-Tuchr \
-Tuid_t \
- -Tuint \
-Tuint16 \
-Tuint32 \
-Tuint64 \
-Tuint8 \
- -Tunknown \
- -Tushort \
-Tva_list \
-Tvarattrib \
- -Tvm_offset_t \
- -Tvm_size_t \
-Tvoid \
-Tvoidp \
-Tvoidpf \
-Twchar_t \
- -Twctype_t \
- -Twint_t \
-Txl_btree_delete \
-Txl_btree_delete_page \
-Txl_btree_insert \
--- 1235,1264 ----
-TsqlparseInfo \
-TsqlparseState \
-Tssize_t \
-Tsubstitute_actual_parameters_context \
-Ttcflag_t \
-TteReqs \
-Ttext \
-Ttime_t \
-Ttimestamp \
+ -Ttlist_vinfo \
-Tts_db_fctx \
-TuInt \
-TuLong \
-Tu_char \
-Tu_int \
-Tuchr \
-Tuid_t \
-Tuint16 \
-Tuint32 \
-Tuint64 \
-Tuint8 \
-Tva_list \
-Tvarattrib \
-Tvoid \
-Tvoidp \
-Tvoidpf \
-Twchar_t \
-Txl_btree_delete \
-Txl_btree_delete_page \
-Txl_btree_insert \
***************
*** 1690,1710 ****
-Txl_btree_newroot \
-Txl_btree_split \
-Txl_btreetid \
-Txl_heap_clean \
-Txl_heap_delete \
-Txl_heap_header \
-Txl_heap_insert \
-Txl_heap_newpage \
-Txl_heap_update \
-Txl_heaptid \
-Txl_seq_rec \
-Txl_smgr_create \
-Txl_smgr_truncate \
-Txl_xact_abort \
-Txl_xact_commit \
-Tyy_size_t \
-Tyy_state_type \
- -Tyysigned_char \
-Tz_stream \
-Tz_streamp \
/tmp/$$a >/tmp/$$ 2>&1
--- 1267,1292 ----
-Txl_btree_newroot \
-Txl_btree_split \
-Txl_btreetid \
+ -Txl_dbase_create_rec \
+ -Txl_dbase_drop_rec \
-Txl_heap_clean \
-Txl_heap_delete \
-Txl_heap_header \
-Txl_heap_insert \
+ -Txl_heap_lock \
-Txl_heap_newpage \
-Txl_heap_update \
-Txl_heaptid \
+ -Txl_multixact_create \
-Txl_seq_rec \
-Txl_smgr_create \
-Txl_smgr_truncate \
+ -Txl_tblspc_create_rec \
+ -Txl_tblspc_drop_rec \
-Txl_xact_abort \
-Txl_xact_commit \
-Tyy_size_t \
-Tyy_state_type \
-Tz_stream \
-Tz_streamp \
/tmp/$$a >/tmp/$$ 2>&1
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-06-28 12:53 ` Andrew Dunstan <andrew@dunslane.net>
3 siblings, 0 replies; 52+ messages in thread
From: Andrew Dunstan @ 2005-06-28 12:53 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Luke, Alon
OK, I'm going to apply the patch to my copy and try to get my head
around it. meanwhile:
. we should not be describing things as "old" or "new". The person
reading the code might have no knowledge of the history, and should not
need to.
. we should not have "slow" and "fast" either. We should have "text",
"csv" and "binary".
IOW, the patch comments look slightly like it is intended for after the
fact application rather than incorporation into the main code.
Are you looking at putting CSV mode into the fast code? Please let me
know if you have questions about that. There are only a few days left to
whip this into shape.
cheers
andrew
Luke Lonergan wrote:
>Andrew,
>
>
>
>>Something strange has happened. I suspect that you've inadvertantly used
>>GNU indent or an unpatched BSD indent. pgindent needs a special patched
>>BSD indent to work according to the PG standards - see the README
>>
>>
>
>OK - phew! I generated new symbols for pgindent and fixed a bug in the awk
>scripting within (diff attached) and ran against the CVS tip copy.c and got
>only minor changes in formatting that appear to be consistent with the rest
>of the code. I pgindent'ed the COPY FROM performance modded code and it
>looks good and tests good.
>
>Only formatting changes to the previous patch for copy.c attached.
>
>Patch to update pgindent with new symbols and fix a bug in an awk section
>(extra \\ in front of a ')').
>
>- Luke
>
>
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-06-28 23:14 ` Bruce Momjian <pgman@candle.pha.pa.us>
3 siblings, 0 replies; 52+ messages in thread
From: Bruce Momjian @ 2005-06-28 23:14 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Luke Lonergan wrote:
> Patch to update pgindent with new symbols and fix a bug in an awk section
> (extra \\ in front of a ')').
Yea, that '\' wasn't needed. I applied the following patch to use //
instead of "" for patterns, and removed the unneeded backslash.
I will update the typedefs in a separate commit.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Attachments:
/bjm/diff (text/plain, 5.3K) download
Index: src/tools/pgindent/pgindent
===================================================================
RCS file: /cvsroot/pgsql/src/tools/pgindent/pgindent,v
retrieving revision 1.73
diff -c -c -r1.73 pgindent
*** src/tools/pgindent/pgindent 7 Oct 2004 14:15:50 -0000 1.73
--- src/tools/pgindent/pgindent 28 Jun 2005 23:12:07 -0000
***************
*** 50,62 ****
if (NR >= 2)
print line1;
if (NR >= 2 &&
! line2 ~ "^{[ ]*$" &&
! line1 !~ "^struct" &&
! line1 !~ "^enum" &&
! line1 !~ "^typedef" &&
! line1 !~ "^extern[ ][ ]*\"C\"" &&
! line1 !~ "=" &&
! line1 ~ "\)")
print "int pgindent_func_no_var_fix;";
line1 = line2;
}
--- 50,62 ----
if (NR >= 2)
print line1;
if (NR >= 2 &&
! line2 ~ /^{[ ]*$/ &&
! line1 !~ /^struct/ &&
! line1 !~ /^enum/ &&
! line1 !~ /^typedef/ &&
! line1 !~ /^extern[ ][ ]*"C"/ &&
! line1 !~ /=/ &&
! line1 ~ /)/)
print "int pgindent_func_no_var_fix;";
line1 = line2;
}
***************
*** 70,77 ****
line2 = $0;
if (skips > 0)
skips--;
! if (line1 ~ "^#ifdef[ ]*__cplusplus" &&
! line2 ~ "^extern[ ]*\"C\"[ ]*$")
{
print line1;
print line2;
--- 70,77 ----
line2 = $0;
if (skips > 0)
skips--;
! if (line1 ~ /^#ifdef[ ]*__cplusplus/ &&
! line2 ~ /^extern[ ]*"C"[ ]*$/)
{
print line1;
print line2;
***************
*** 81,88 ****
line2 = "";
skips = 2;
}
! else if (line1 ~ "^#ifdef[ ]*__cplusplus" &&
! line2 ~ "^}[ ]*$")
{
print line1;
print "/* Close extern \"C\" */";
--- 81,88 ----
line2 = "";
skips = 2;
}
! else if (line1 ~ /^#ifdef[ ]*__cplusplus/ &&
! line2 ~ /^}[ ]*$/)
{
print line1;
print "/* Close extern \"C\" */";
***************
*** 1732,1738 ****
# work around misindenting of function with no variables defined
awk '
{
! if ($0 ~ "^[ ]*int[ ]*pgindent_func_no_var_fix;")
{
if (getline && $0 != "")
print $0;
--- 1732,1738 ----
# work around misindenting of function with no variables defined
awk '
{
! if ($0 ~ /^[ ]*int[ ]*pgindent_func_no_var_fix;/)
{
if (getline && $0 != "")
print $0;
***************
*** 1751,1759 ****
# line3 = $0;
# if (skips > 0)
# skips--;
! # if (line1 ~ " *{$" &&
! # line2 ~ " *[^;{}]*;$" &&
! # line3 ~ " *}$")
# {
# print line2;
# line2 = "";
--- 1751,1759 ----
# line3 = $0;
# if (skips > 0)
# skips--;
! # if (line1 ~ / *{$/ &&
! # line2 ~ / *[^;{}]*;$/ &&
! # line3 ~ / *}$/)
# {
# print line2;
# line2 = "";
***************
*** 1778,1786 ****
line3 = $0;
if (skips > 0)
skips--;
! if (line1 ~ " *{$" &&
! line2 ~ "^$" &&
! line3 ~ " */\\*$")
{
print line1;
print line3;
--- 1778,1786 ----
line3 = $0;
if (skips > 0)
skips--;
! if (line1 ~ / *{$/ &&
! line2 ~ /^$/ &&
! line3 ~ / *\/\*$/)
{
print line1;
print line3;
***************
*** 1819,1826 ****
line2 = $0;
if (skips > 0)
skips--;
! if (line1 ~ "^$" &&
! line2 ~ "^#endif")
{
print line2;
line2 = "";
--- 1819,1826 ----
line2 = $0;
if (skips > 0)
skips--;
! if (line1 ~ /^$/ &&
! line2 ~ /^#endif/)
{
print line2;
line2 = "";
***************
*** 1844,1850 ****
line1 = line2;
}
END {
! if (NR >= 1 && line2 ~ "^#endif")
printf "\n";
print line1;
}' |
--- 1844,1850 ----
line1 = line2;
}
END {
! if (NR >= 1 && line2 ~ /^#endif/)
printf "\n";
print line1;
}' |
***************
*** 1853,1868 ****
# like real functions.
awk ' BEGIN {paren_level = 0}
{
! if ($0 ~ /^[a-zA-Z_][a-zA-Z_0-9]*[^\(]*$/)
{
saved_len = 0;
saved_lines[++saved_len] = $0;
if ((getline saved_lines[++saved_len]) == 0)
print saved_lines[1];
else
! if (saved_lines[saved_len] !~ /^[a-zA-Z_][a-zA-Z_0-9]*\(/ ||
! saved_lines[saved_len] ~ /^[a-zA-Z_][a-zA-Z_0-9]*\(.*\)$/ ||
! saved_lines[saved_len] ~ /^[a-zA-Z_][a-zA-Z_0-9]*\(.*\);$/)
{
print saved_lines[1];
print saved_lines[2];
--- 1853,1868 ----
# like real functions.
awk ' BEGIN {paren_level = 0}
{
! if ($0 ~ /^[a-zA-Z_][a-zA-Z_0-9]*[^(]*$/)
{
saved_len = 0;
saved_lines[++saved_len] = $0;
if ((getline saved_lines[++saved_len]) == 0)
print saved_lines[1];
else
! if (saved_lines[saved_len] !~ /^[a-zA-Z_][a-zA-Z_0-9]*(/ ||
! saved_lines[saved_len] ~ /^[a-zA-Z_][a-zA-Z_0-9]*(.*)$/ ||
! saved_lines[saved_len] ~ /^[a-zA-Z_][a-zA-Z_0-9]*(.*);$/)
{
print saved_lines[1];
print saved_lines[2];
***************
*** 1879,1885 ****
}
for (i=1; i <= saved_len; i++)
{
! if (i == 1 && saved_lines[saved_len] ~ /\);$/)
{
printf "%s", saved_lines[i];
if (substr(saved_lines[i], length(saved_lines[i]),1) != "*")
--- 1879,1885 ----
}
for (i=1; i <= saved_len; i++)
{
! if (i == 1 && saved_lines[saved_len] ~ /);$/)
{
printf "%s", saved_lines[i];
if (substr(saved_lines[i], length(saved_lines[i]),1) != "*")
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-15 00:22 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 15:23 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
3 siblings, 3 replies; 52+ messages in thread
From: Alon Goldshuv @ 2005-07-15 00:22 UTC (permalink / raw)
To: pgsql-patches@postgresql.org
I revisited my patch and removed the code duplications that were there, and
added support for CSV with buffered input, so CSV now runs faster too
(although it is not as optimized as the TEXT format parsing). So now
TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
Patch attached.
Greetings,
Alon.
Attachments:
copy_parse_improvements_V15.patch (application/octet-stream, 78.0K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.247
diff -c -r1.247 copy.c
*** src/backend/commands/copy.c 10 Jul 2005 21:13:58 -0000 1.247
--- src/backend/commands/copy.c 15 Jul 2005 00:07:49 -0000
***************
*** 8,14 ****
*
*
* IDENTIFICATION
! * $PostgreSQL: pgsql/src/backend/commands/copy.c,v 1.247 2005/07/10 21:13:58 tgl Exp $
*
*-------------------------------------------------------------------------
*/
--- 8,14 ----
*
*
* IDENTIFICATION
! * $PostgreSQL: pgsql/src/backend/commands/copy.c,v 1.246 2005/06/28 05:08:53 tgl Exp $
*
*-------------------------------------------------------------------------
*/
***************
*** 50,55 ****
--- 50,56 ----
#define ISOCTAL(c) (((c) >= '0') && ((c) <= '7'))
#define OCTVALUE(c) ((c) - '0')
+ #define COPY_BUF_SIZE 65536
/*
* Represents the different source/dest cases we need to worry about at
***************
*** 63,78 ****
} CopyDest;
/*
- * State indicator showing what stopped CopyReadAttribute()
- */
- typedef enum CopyReadResult
- {
- NORMAL_ATTR,
- END_OF_LINE,
- UNTERMINATED_FIELD
- } CopyReadResult;
-
- /*
* Represents the end-of-line terminator type of the input
*/
typedef enum EolType
--- 64,69 ----
***************
*** 97,102 ****
--- 88,96 ----
static EolType eol_type; /* EOL type of input */
static int client_encoding; /* remote side's character encoding */
static int server_encoding; /* local encoding */
+ static char eol_ch[2]; /* The byte values of the 1 or 2 eol bytes */
+ static bool client_encoding_only; /* true if client encoding is a non
+ * supported server encoding */
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
***************
*** 104,126 ****
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attribute_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attribute_buf's data buffer!
*/
! static StringInfoData attribute_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attribute_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
--- 98,141 ----
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
+ /*
+ * Static variables for buffered input parsing
+ */
+ static char input_buf[COPY_BUF_SIZE + 1]; /* extra byte for '\0' */
+ static int buffer_index; /* input buffer index */
+ static bool end_marker;
+ static char *begloc;
+ static char *endloc;
+ static bool buf_done; /* finished processing the current buffer */
+ static bool line_done; /* finished processing the whole line or
+ * stopped in the middle */
+ /* these are for CSV format */
+ static bool in_quote;
+ static bool last_was_esc;
+
+ /* these are for TEXT format */
+ static bool esc_in_prevbuf; /* escape was last character of the data
+ * input buffer */
+ static bool cr_in_prevbuf; /* CR was last character of the data input
+ * buffer */
+
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attr_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attr_buf's data buffer!
*/
! static StringInfoData attr_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attr_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
***************
*** 137,155 ****
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char * quote, char * escape);
! static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
! static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, int32 typmod, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
--- 152,176 ----
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLineText(size_t bytesread, char *escape);
! static bool CopyReadLineCSV(size_t bytesread, char *quote, char *escape);
! static void CopyReadAttributesText(const char *delim, const char *escape, const char *null_print,
! int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr);
! static void CopyReadAttributesCSV(const char *delim, const char *null_print, char *quote,
! char *escape, int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, bool *isnull);
! static char *CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
! bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
+ static bool DetectLineEnd(size_t bytesread, char *quote, char *escape);
+
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
***************
*** 159,175 ****
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static void CopyGetData(void *databuf, int datasize);
! static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
- static int CopyPeekChar(void);
- static void CopyDonePeek(int c, bool pickup);
static void CopySendInt32(int32 val);
static int32 CopyGetInt32(void);
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
/*
* Send copy start/stop messages for frontend copies. These have changed
--- 180,197 ----
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static int CopyGetData(void *databuf, int datasize);
#define CopyGetEof() (fe_eof)
static void CopySendInt32(int32 val);
static int32 CopyGetInt32(void);
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
+ /* byte scaning utils */
+ static char *scanTextLine(const char *s, char c, size_t len);
+ static char *scanCSVLine(const char *s, char c1, char c2, char c3, size_t len);
+ static char *scanTextAttr(const char *s, char c1, char c2, size_t len);
/*
* Send copy start/stop messages for frontend copies. These have changed
***************
*** 382,395 ****
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
*/
! static void
CopyGetData(void *databuf, int datasize)
{
switch (copy_dest)
{
case COPY_FILE:
! fread(databuf, datasize, 1, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
--- 404,422 ----
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
+ *
+ * Returns: the number of bytes that were successfully read
+ * into the data buffer.
*/
! static int
CopyGetData(void *databuf, int datasize)
{
+ size_t bytesread = 0;
+
switch (copy_dest)
{
case COPY_FILE:
! bytesread = fread(databuf, 1, datasize, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
***************
*** 401,406 ****
--- 428,435 ----
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
+ bytesread += datasize; /* update the count of bytes that were
+ * read so far */
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
***************
*** 429,435 ****
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
--- 458,464 ----
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return bytesread;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
***************
*** 459,598 ****
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
datasize -= avail;
}
break;
}
- }
-
- static int
- CopyGetChar(void)
- {
- int ch;
-
- switch (copy_dest)
- {
- case COPY_FILE:
- ch = getc(copy_file);
- break;
- case COPY_OLD_FE:
- ch = pq_getbyte();
- if (ch == EOF)
- {
- /* Only a \. terminator is legal EOF in old protocol */
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("unexpected EOF on client connection")));
- }
- break;
- case COPY_NEW_FE:
- {
- unsigned char cc;
-
- CopyGetData(&cc, 1);
- if (fe_eof)
- ch = EOF;
- else
- ch = cc;
- break;
- }
- default:
- ch = EOF;
- break;
- }
- if (ch == EOF)
- fe_eof = true;
- return ch;
- }
-
- /*
- * CopyPeekChar reads a byte in "peekable" mode.
- *
- * after each call to CopyPeekChar, a call to CopyDonePeek _must_
- * follow, unless EOF was returned.
- *
- * CopyDonePeek will either take the peeked char off the stream
- * (if pickup is true) or leave it on the stream (if pickup is false).
- */
- static int
- CopyPeekChar(void)
- {
- int ch;
-
- switch (copy_dest)
- {
- case COPY_FILE:
- ch = getc(copy_file);
- break;
- case COPY_OLD_FE:
- ch = pq_peekbyte();
- if (ch == EOF)
- {
- /* Only a \. terminator is legal EOF in old protocol */
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("unexpected EOF on client connection")));
- }
- break;
- case COPY_NEW_FE:
- {
- unsigned char cc;
-
- CopyGetData(&cc, 1);
- if (fe_eof)
- ch = EOF;
- else
- ch = cc;
- break;
- }
- default:
- ch = EOF;
- break;
- }
- if (ch == EOF)
- fe_eof = true;
- return ch;
- }
-
- static void
- CopyDonePeek(int c, bool pickup)
- {
- if (fe_eof)
- return; /* can't unget an EOF */
- switch (copy_dest)
- {
- case COPY_FILE:
- if (!pickup)
- {
- /* We don't want to pick it up - so put it back in there */
- ungetc(c, copy_file);
- }
- /* If we wanted to pick it up, it's already done */
- break;
- case COPY_OLD_FE:
- if (pickup)
- {
- /* We want to pick it up */
- (void) pq_getbyte();
- }
! /*
! * If we didn't want to pick it up, just leave it where it
! * sits
! */
! break;
! case COPY_NEW_FE:
! if (!pickup)
! {
! /* We don't want to pick it up - so put it back in there */
! copy_msgbuf->cursor--;
! }
! /* If we wanted to pick it up, it's already done */
! break;
! }
}
-
/*
* These functions do apply some data conversion
*/
--- 488,503 ----
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
+ bytesread += avail; /* update the count of bytes that were
+ * read so far */
datasize -= avail;
}
break;
}
! return bytesread;
}
/*
* These functions do apply some data conversion
*/
***************
*** 968,980 ****
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
--- 873,905 ----
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attr_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
+ /*
+ * check if the client encoding is one of the 5 encodings
+ * that are not supported as a server encodings.
+ */
+ switch (client_encoding)
+ {
+ case PG_SJIS:
+ case PG_BIG5:
+ case PG_GBK:
+ case PG_UHC:
+ case PG_GB18030:
+ client_encoding_only = true;
+ break;
+ default:
+ client_encoding_only = false;
+ }
+
+ if(!csv_mode)
+ escape = "\\"; /* default for text format */
+
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
***************
*** 1105,1113 ****
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attribute_buf.data);
pfree(line_buf.data);
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
--- 1030,1039 ----
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attr_buf.data);
pfree(line_buf.data);
+
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
***************
*** 1345,1351 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
--- 1271,1278 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
***************
*** 1395,1404 ****
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attribute_buf.data);
}
else
{
--- 1322,1331 ----
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attr_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attr_buf.data);
}
else
{
***************
*** 1406,1411 ****
--- 1333,1340 ----
if (line_buf_converted ||
client_encoding == server_encoding)
{
+ /* Strip off the newline */
+ *(line_buf.data + line_buf.len - 1) = '\0';
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
copy_relname, copy_lineno,
***************
*** 1485,1491 ****
Oid in_func_oid;
Datum *values;
char *nulls;
- bool done = false;
bool isnull;
ResultRelInfo *resultRelInfo;
EState *estate = CreateExecutorState(); /* for ExecConstraints() */
--- 1414,1419 ----
***************
*** 1496,1501 ****
--- 1424,1434 ----
ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
+ int *attr_offsets;
+ int null_print_len; /* length of null print */
+ bool no_more_data;
+ ListCell *cur;
+
tupDesc = RelationGetDescr(rel);
attr = tupDesc->attrs;
***************
*** 1607,1613 ****
}
}
! /* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
/*
--- 1540,1548 ----
}
}
! /*
! * Prepare to catch AFTER triggers.
! */
AfterTriggerBeginQuery();
/*
***************
*** 1671,1676 ****
--- 1606,1612 ----
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
+ attr_offsets = (int *) palloc(num_phys_attrs * sizeof(int));
/* Make room for a PARAM_EXEC value for domain constraint checks */
if (hasConstraints)
***************
*** 1691,1712 ****
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /* on input just throw the header line away */
! if (header_line)
{
! copy_lineno++;
! done = CopyReadLine(quote, escape) ;
}
! while (!done)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;
CHECK_FOR_INTERRUPTS();
- copy_lineno++;
-
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
--- 1627,1681 ----
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /*
! * initialize buffered scan variables.
! */
! if(csv_mode)
! {
! in_quote = false;
! last_was_esc = false;
! }
!
! null_print_len = strlen(null_print);
!
! /* Set up data buffer to hold a chunk of data */
! MemSet(input_buf, ' ', COPY_BUF_SIZE * sizeof(char));
! input_buf[COPY_BUF_SIZE] = '\0';
!
! no_more_data = false; /* no more input data to read from file or FE */
! line_done = true;
! buf_done = false;
!
! do
{
! size_t bytesread = 0;
!
! /* read a chunk of data into the buffer */
! if (!binary)
! {
! bytesread = CopyGetData(input_buf, COPY_BUF_SIZE);
! buf_done = false;
!
! /* set buffer pointers to beginning of the buffer */
! begloc = input_buf;
! buffer_index = 0;
}
! /*
! * continue if some bytes were read or if we didn't reach EOF. if we
! * both reached EOF _and_ no bytes were read, quit the loop we are
! * done
! */
! if (bytesread > 0 || !fe_eof || binary)
! {
!
! while (!buf_done)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;
CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
***************
*** 1716,1746 ****
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
if (!binary)
{
- CopyReadResult result = NORMAL_ATTR;
- char *string;
- ListCell *cur;
/* Actually read the line into memory here */
! done = csv_mode ?
! CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed
! * by EOF, ie, process the line and then exit loop on next
! * iteration.
*/
! if (done && line_buf.len == 0)
break;
if (file_has_oids)
{
/* can't be in CSV mode here */
! string = CopyReadAttribute(delim, null_print,
! &result, &isnull);
if (isnull)
ereport(ERROR,
--- 1685,1729 ----
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
+ /* reset attribute pointers */
+ MemSet(attr_offsets, 0, num_phys_attrs * sizeof(int));
if (!binary)
{
/* Actually read the line into memory here */
! line_done = csv_mode ?
! CopyReadLineCSV(bytesread, quote, escape) :
! CopyReadLineText(bytesread, escape);
/*
! * if finished processing data line - increment line count.
! * Otherwise, if eof is not yet reached, we skip att parsing
! * and read more data. But if eof _was_ reached it means
! * that the original last data line is defective and
! * we want to catch that error later on.
*/
! if (line_done)
! copy_lineno++;
! else if (!fe_eof || end_marker )
break;
+ if (header_line)
+ {
+ line_buf.len = 0; /* we can reset line buffer now. */
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ header_line = false;
+ continue;
+ }
+
if (file_has_oids)
{
+ char *oid_string;
+
/* can't be in CSV mode here */
! oid_string = CopyReadOidAttr(delim, null_print, null_print_len,
! &isnull);
if (isnull)
ereport(ERROR,
***************
*** 1750,1756 ****
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 1733,1739 ----
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(oid_string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 1759,1792 ****
}
}
! /* Loop to read the user attributes on the line. */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
! /*
! * If prior attr on this line was ended by newline,
! * complain.
! */
! if (result != NORMAL_ATTR)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! if (csv_mode)
! {
! string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
! if (result == UNTERMINATED_FIELD)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
! }
else
! string = CopyReadAttribute(delim, null_print,
! &result, &isnull);
if (csv_mode && isnull && force_notnull[m])
{
--- 1742,1770 ----
}
}
! /* parse all the attribute in the data line */
! if(csv_mode)
! CopyReadAttributesCSV(delim, null_print, quote, escape, null_print_len,
! nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
! else
! CopyReadAttributesText(delim, escape, null_print, null_print_len,
! nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
!
! /*
! * Loop to read the user attributes on the line.
! */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
+ char *string;
! string = attr_buf.data + attr_offsets[m];
! if (nulls[m] == ' ')
! isnull = false;
else
! isnull = true;
if (csv_mode && isnull && force_notnull[m])
{
***************
*** 1806,1823 ****
copy_attname = NULL;
}
}
-
- /*
- * Complain if there are more fields on the input line.
- *
- * Special case: if we're reading a zero-column table, we won't
- * yet have called CopyReadAttribute() at all; so no error if
- * line is empty.
- */
- if (result == NORMAL_ATTR && line_buf.len != 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
}
else
{
--- 1784,1789 ----
***************
*** 1828,1834 ****
fld_count = CopyGetInt16();
if (CopyGetEof() || fld_count == -1)
{
! done = true;
break;
}
--- 1794,1801 ----
fld_count = CopyGetInt16();
if (CopyGetEof() || fld_count == -1)
{
! buf_done = true;
! no_more_data = true;
break;
}
***************
*** 1845,1851 ****
DatumGetObjectId(CopyReadBinaryAttribute(0,
&oid_in_function,
oid_typioparam,
- -1,
&isnull));
if (isnull || loaded_oid == InvalidOid)
ereport(ERROR,
--- 1812,1817 ----
***************
*** 1865,1871 ****
values[m] = CopyReadBinaryAttribute(i,
&in_functions[m],
typioparams[m],
- attr[m]->atttypmod,
&isnull);
nulls[m] = isnull ? 'n' : ' ';
copy_attname = NULL;
--- 1831,1836 ----
***************
*** 1885,1891 ****
nulls[defmap[i]] = ' ';
}
! /* Next apply any domain constraints */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
--- 1850,1858 ----
nulls[defmap[i]] = ' ';
}
! /*
! * Next apply any domain constraints
! */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
***************
*** 1912,1924 ****
}
}
! /* And now we can form the input tuple. */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /* Triggers and stuff need to be invoked in query context. */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
--- 1879,1895 ----
}
}
! /*
! * And now we can form the input tuple.
! */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /*
! * Triggers and stuff need to be invoked in query context.
! */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
***************
*** 1945,1955 ****
/* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /* Check the constraints of the tuple */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /* OK, store the tuple and create index entries for it */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
--- 1916,1930 ----
/* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /*
! * Check the constraints of the tuple
! */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /*
! * OK, store the tuple and create index entries for it
! */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
***************
*** 1958,1976 ****
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /* Done, clean up */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /* Execute AFTER STATEMENT insertion triggers */
ExecASInsertTriggers(estate, resultRelInfo);
! /* Handle queued AFTER triggers */
AfterTriggerEndQuery(estate);
pfree(values);
pfree(nulls);
--- 1933,1970 ----
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
+
+ line_buf.len = 0; /* we can reset line buffer now. */
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ } /* end while(!buf_done) */
+ } /* end if (bytesread > 0 || !fe_eof) */
+ else
+ /* no bytes read, end of data */
+ {
+ no_more_data = true;
}
+ } while (!no_more_data);
! /*
! * Done, clean up
! */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
ExecASInsertTriggers(estate, resultRelInfo);
! /*
! * Handle queued AFTER triggers
! */
AfterTriggerEndQuery(estate);
+ pfree(attr_offsets);
+
pfree(values);
pfree(nulls);
***************
*** 1990,2262 ****
/*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
*/
static bool
! CopyReadLine(char * quote, char * escape)
{
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
! unsigned char s[2];
char *cvt;
! bool in_quote = false, last_was_esc = false, csv_mode = false;
! char quotec = '\0', escapec = '\0';
!
! if (quote)
! {
! csv_mode = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
!
!
! s[1] = 0;
!
! /* reset line_buf to empty */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
/* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /* set default status */
! result = false;
/*
! * In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
! *
! * In Text mode, for backwards compatibility we allow
! * backslashes to escape newline characters. Backslashes other than
! * the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
! *
! * In CSV mode, CR and NL inside q quoted field are just part of the
! * data value and are put in line_buf. We keep just enough state
! * to know if we are currently in a quoted field or not.
! *
! * These four characters, and only these four, are assumed the same in
! * frontend and backend encodings.
! *
! * We do not assume that second and later bytes of a frontend
! * multibyte character couldn't look like ASCII characters.
*/
! for (;;)
{
! c = CopyGetChar();
! if (c == EOF)
{
! result = true;
! break;
}
! if (csv_mode)
{
/*
! * Dealing with quotes and escapes here is mildly tricky. If the
! * quote char is also the escape char, there's no problem - we
! * just use the char as a toggle. If they are different, we need
! * to ensure that we only take account of an escape inside a quoted
! * field and immediately preceding a quote char, and not the
! * second in a escape-escape sequence.
*/
! if (in_quote && c == escapec)
! last_was_esc = ! last_was_esc;
! if (c == quotec && ! last_was_esc)
! in_quote = ! in_quote;
! if (c != escapec)
! last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
! * the best we can do.
*/
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
! }
! if (!in_quote && c == '\r')
{
! if (eol_type == EOL_NL)
{
! if (! csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
! /* Check for \r\n on first line, _and_ handle \r\n. */
! if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
! int c2 = CopyPeekChar();
! if (c2 == '\n')
! {
! CopyDonePeek(c2, true); /* eat newline */
! eol_type = EOL_CRNL;
}
else
{
! /* found \r, but no \n */
if (eol_type == EOL_CRNL)
{
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
!
}
/*
! * if we got here, it is the first line and we didn't
! * get \n, so put it back
*/
! CopyDonePeek(c2, false);
! eol_type = EOL_CR;
}
}
- break;
}
! if (!in_quote && c == '\n')
{
! if (eol_type == EOL_CR || eol_type == EOL_CRNL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal newline found in data"),
! errhint("Use \"\\n\" to represent newline.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
}
- eol_type = EOL_NL;
- break;
- }
! if ((line_buf.len == 0 || !csv_mode) && c == '\\')
! {
! int c2;
!
! if (csv_mode)
! c2 = CopyPeekChar();
! else
! c2 = c = CopyGetChar();
! if (c2 == EOF)
{
! result = true;
! if (csv_mode)
! CopyDonePeek(c2, true);
break;
}
! if (c2 == '.')
! {
! if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
! if (eol_type == EOL_CRNL)
! {
! c = CopyGetChar();
! if (c == '\n')
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
! if (c != '\r')
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker corrupt")));
}
- c = CopyGetChar();
- if (c != '\r' && c != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("end-of-copy marker corrupt")));
- if ((eol_type == EOL_NL && c != '\n') ||
- (eol_type == EOL_CRNL && c != '\n') ||
- (eol_type == EOL_CR && c != '\r'))
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("end-of-copy marker does not match previous newline style")));
/*
! * In protocol version 3, we should ignore anything after
! * \. up to the protocol end of copy data. (XXX maybe
! * better not to treat \. as special?)
*/
! if (copy_dest == COPY_NEW_FE)
{
! while (c != EOF)
! c = CopyGetChar();
}
! result = true; /* report EOF */
break;
}
! if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /* not EOF mark, so emit \ and following char literally */
! appendStringInfoCharMacro(&line_buf, '\\');
}
! appendStringInfoCharMacro(&line_buf, c);
/*
! * When client encoding != server, must be careful to read the
! * extra bytes of a multibyte character exactly, since the
! * encoding might not ensure they don't look like ASCII. When the
! * encodings are the same, we need not do this, since no server
! * encoding we use has ASCII-like following bytes.
*/
! if (change_encoding)
! {
! s[0] = c;
! mblen = pg_encoding_mblen(client_encoding, s);
! for (j = 1; j < mblen; j++)
! {
! c = CopyGetChar();
! if (c == EOF)
{
! result = true;
break;
}
- appendStringInfoCharMacro(&line_buf, c);
}
! if (result)
! break; /* out of outer loop */
}
- } /* end of outer loop */
! /* Done reading the line. Convert it to server encoding. */
! if (change_encoding)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
--- 1984,2422 ----
/*
! * Finds the next TEXT line that is in the input buffer and loads
! * it into line_buf. Returns an indication if the line that was read
! * is complete (if an unescaped line-end was encountered). If we
! * reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
*/
static bool
! CopyReadLineText(size_t bytesread, char *escape)
{
! int linesize;
! bool transcode = (client_encoding != server_encoding);
char *cvt;
! char escapec = '\0';
/* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /*
! * set the escape char for text format ('\\' by default).
! */
! escapec = escape[0];
/*
! * Detect end of line type if not already detected.
*/
! if (eol_type == EOL_UNKNOWN)
{
! bool eol_detected = DetectLineEnd(bytesread, NULL, escape);
!
! if (!eol_detected)
{
! /* load entire input buffer into line buf, and quit */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! buf_done = true;
! return false;
! }
}
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /* if we started scanning from the 1st byte of the buffer */
! if (begloc == input_buf)
! {
! /* and had a CR in last byte of prev buf */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence
! * (linefeed)
! */
! if (*begloc == eol_ch[1])
{
/*
! * load that one linefeed byte and indicate we are done
! * with the data line
*/
+ appendBinaryStringInfo(&line_buf, begloc, 1);
+ buffer_index++;
+ begloc++;
+ esc_in_prevbuf = false;
+ cr_in_prevbuf = false;
+ return true;
+ }
+ }
! cr_in_prevbuf = false;
! }
! }
/*
! * (we need a loop so that if eol_ch is found, but prev ch is backslash,
! * we can search for the next eol_ch)
*/
! while (true)
! {
! /* reached end of buffer */
! if ((endloc = scanTextLine(begloc, eol_ch[0], bytesread - buffer_index)) == NULL)
! {
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! if (line_buf.len > 1)
{
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == escapec)
{
! esc_in_prevbuf = true;
!
! if (line_buf.len > 2)
! {
! last_ch--;
! if (*last_ch == escapec)
! esc_in_prevbuf = false;
}
! }
! else if (*last_ch == '\r')
{
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
! }
! }
! line_done = false;
! buf_done = true;
! break;
}
else
+ /* found the 1st eol ch in input_buf. */
{
! bool eol_found = true;
! bool eol_escaped = true;
!
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
if (eol_type == EOL_CRNL)
{
! /* check if there is a '\n' after the '\r' */
! if (*(endloc + 1) == '\n')
! {
! /* this is a line end */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
else
! /* just a CR, not a line end */
! eol_found = false;
}
/*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
*/
! if (buffer_index >= bytesread)
! buf_done = true;
!
! /*
! * Check if the 1st end of line ch is escaped.
! */
! if (endloc != input_buf) /* can we look 1 char back? */
! {
! if (*(endloc - 1) != escapec) /* prev char is not an escape */
! eol_escaped = false;
! else
! /* prev char is an escape */
! {
! if (endloc != (input_buf + 1)) /* can we look another
! * char back? */
! {
! /* it's a double escape char, so it's not an escape */
! if (*(endloc - 2) == escapec)
! eol_escaped = false;
! /* else it's a single escape char, so EOL is ascaped */
! }
! else
! {
! /* we need to check in the last buffer */
! if (esc_in_prevbuf)
! eol_escaped = false;
}
}
}
! else
! /*
! * this eol ch is first ch in buffer, check for escape in prev buf
! */
{
! if (!esc_in_prevbuf)
! eol_escaped = false;
}
! esc_in_prevbuf = false; /* reset variable */
! /*
! * if eol was found, and it isn't escaped, line is done
! */
! if ((eol_escaped == false) && eol_found)
{
! line_done = true;
break;
}
! else
! /* stay in the loop and process some more data. */
! line_done = false;
!
! } /* end of found eol_ch */
! }
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
! {
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /* transfer converted data back to line_buf */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
! }
!
! /* indicate that conversion had occured */
! line_buf_converted = true;
!
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
!
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
! {
! fe_eof = true;
! /* we don't want to process a \. as data line, want to quit. */
! line_done = false;
! buf_done = true;
! }
!
! return line_done;
! }
!
! /*
! * Finds the next CSV line that is in the input buffer and loads
! * it into line_buf. Returns an indication if the line that was read
! * is complete (if an unescaped line-end was encountered). If we
! * reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
! */
! static bool
! CopyReadLineCSV(size_t bytesread, char *quote, char *escape)
! {
! int linesize;
! char *cvt;
! char quotec = '\0',
! escapec = '\0';
! bool transcode = (client_encoding != server_encoding);
!
! /* mark that encoding conversion hasn't occurred yet */
! line_buf_converted = false;
!
! escapec = escape[0];
! quotec = quote[0];
!
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
!
! /*
! * Detect end of line type if not already detected.
! */
! if (eol_type == EOL_UNKNOWN)
! {
! bool eol_detected = DetectLineEnd(bytesread, quote, escape);
!
! if (!eol_detected)
! {
! /* load entire input buffer into line buf, and quit */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! line_done = false;
! buf_done = true;
!
! return line_done;
! }
! }
!
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /* if we started scanning from the 1st byte of the buffer */
! if (begloc == input_buf)
! {
! /* and had a CR in last byte of prev buf */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence
! * (linefeed)
! */
! if (*begloc == eol_ch[1])
! {
! /*
! * load that one linefeed byte and indicate we are done
! * with the data line
! */
! appendBinaryStringInfo(&line_buf, begloc, 1);
! buffer_index++;
! begloc++;
!
! line_done = true;
! esc_in_prevbuf = false;
! cr_in_prevbuf = false;
!
! return line_done;
! }
! }
!
! cr_in_prevbuf = false;
! }
}
/*
! * (we need a loop so that if eol_ch is found, but we are in quotes,
! * we can search for the next eol_ch)
*/
! while (true)
! {
! /* reached end of buffer */
! if ((endloc = scanCSVLine(begloc, eol_ch[0], escapec, quotec, bytesread - buffer_index)) == NULL)
{
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
!
! if (last_was_esc)
! esc_in_prevbuf = true;
!
! if (line_buf.len > 1)
! {
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == '\r')
! {
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
}
! }
!
! line_done = false;
! buf_done = true;
break;
}
+ else
+ /* found 1st eol char in input_buf. */
+ {
+ bool eol_found = true;
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
! /* end of line only if not in quotes */
! if(in_quote)
! {
! line_done = false;
! /* buf done, but still in quote */
! if (buffer_index >= bytesread)
! {
! buf_done = true;
! break;
! }
! }
else
! {
! /* if dos eol, check for '\n' after the '\r' */
! if (eol_type == EOL_CRNL)
! {
! if (*(endloc + 1) == '\n')
! {
! /* this is a line end */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
! else
! /* just a CR, not a line end */
! eol_found = false;
}
! /*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
! */
! if (buffer_index >= bytesread)
! buf_done = true;
/*
! * if eol was found line is done
*/
! if (eol_found)
{
! line_done = true;
break;
}
}
! } /* end of found eol_ch */
}
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
***************
*** 2265,2285 ****
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* Now it's safe to use the buffer in error messages */
line_buf_converted = true;
! return result;
}
/*
* Return decimal value for a hexadecimal digit
*/
static
! int GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
--- 2425,2478 ----
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* indicate that conversion had occured */
line_buf_converted = true;
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
!
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
! {
! fe_eof = true;
! /* we don't want to process a \. as data line, want to quit. */
! line_done = false;
! buf_done = true;
! }
!
! return line_done;
}
/*
* Return decimal value for a hexadecimal digit
*/
static
! int
! GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
***************
*** 2287,2344 ****
return tolower(hex) - 'a' + 10;
}
! /*----------
! * Read the value of a single attribute, performing de-escaping as needed.
! *
! * delim is the column delimiter string (must be just one byte for now).
! * null_print is the null marker string. Note that this is compared to
! * the pre-de-escaped input string.
! *
! * *result is set to indicate what terminated the read:
! * NORMAL_ATTR: column delimiter
! * END_OF_LINE: end of line
! * In either case, the string read up to the terminator is returned.
! *
! * *isnull is set true or false depending on whether the input matched
! * the null marker. Note that the caller cannot check this since the
! * returned string will be the post-de-escaping equivalent, which may
! * look the same as some valid data string.
! *----------
*/
! static char *
! CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull)
{
char c;
! char delimc = delim[0];
! int start_cursor = line_buf.cursor;
! int end_cursor;
! int input_len;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! /* set default status */
! *result = END_OF_LINE;
! for (;;)
{
! end_cursor = line_buf.cursor;
! if (line_buf.cursor >= line_buf.len)
! break;
! c = line_buf.data[line_buf.cursor++];
! if (c == delimc)
{
! *result = NORMAL_ATTR;
! break;
}
! if (c == '\\')
{
! if (line_buf.cursor >= line_buf.len)
! break;
! c = line_buf.data[line_buf.cursor++];
! switch (c)
{
case '0':
case '1':
--- 2480,2769 ----
return tolower(hex) - 'a' + 10;
}
! /*
! * Detected the eol type by looking at the first data row.
! * Possible eol types are NL, CR, or CRNL. If eol type was
! * detected, it is set and a boolean true is returned to
! * indicated detection was successful. If the first data row
! * is longer than the input buffer, we return false and will
! * try again in the next buffer.
*/
! static bool
! DetectLineEnd(size_t bytesread, char *quote, char *escape)
{
+ int index = 0;
char c;
! char quotec = '\0',
! escapec = '\0';
! bool csv = false;
! if (quote) /* CSV format */
! {
! csv = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
! while (index < COPY_BUF_SIZE)
! {
! c = input_buf[index];
! if(csv)
{
! if (in_quote && c == escapec)
! last_was_esc = !last_was_esc;
! if (c == quotec && !last_was_esc)
! in_quote = !in_quote;
! if (c != escapec)
! last_was_esc = false;
! }
!
! if (c == '\n')
{
! if(!csv || (csv && !in_quote))
! {
! eol_type = EOL_NL;
! eol_ch[0] = '\n';
! eol_ch[1] = '\0';
!
! in_quote = false;
! last_was_esc = false;
! return true;
! }
}
! if (c == '\r')
{
! if(!csv || (csv && !in_quote))
! {
! if (input_buf[index + 1] == '\n') /* always safe */
! {
! eol_type = EOL_CRNL;
! eol_ch[0] = '\r';
! eol_ch[1] = '\n';
! }
! else
! {
! eol_type = EOL_CR;
! eol_ch[0] = '\r';
! eol_ch[1] = '\0';
! }
!
! in_quote = false;
! last_was_esc = false;
! return true;
! }
! }
!
! index++;
! }
!
! return false;
! }
!
! /*
! * Read all TEXT attributes. Attributes are parsed from line_buf and
! * inserted (all at once) to attr_buf, while saving pointers to
! * each attribute's starting position.
! *
! * When this routine finishes execution both the nulls array and
! * the attr_offsets array are updated. The attr_offsets will include
! * the offset from the beginning of the attribute array of which
! * each attribute begins. If a specific attribute is not used for this
! * COPY command (ommitted from the column list), a value of 0 will be assigned.
! * For example: for table foo(a,b,c,d,e) and COPY foo(a,b,e)
! * attr_offsets may look something like this after this routine
! * returns: [0,20,0,0,55]. That means that column "a" value starts
! * at byte offset 0, "b" in 20 and "e" in 55, in attr_buf.
! *
! * In the attribute buffer (attr_buf) each attribute
! * is terminated with a '\0', and therefore by using the attr_offsets
! * array we could point to a beginning of an attribute and have it
! * behave as a C string, much like previously done in COPY.
! *
! * Another aspect to improving performance is reducing the frequency
! * of data load into buffers. The original COPY read attribute code
! * loaded a character at a time. In here we try to load a chunk of data
! * at a time. Usually a chunk will include a full data row
! * (unless we have an escaped delim). That effectively reduces the number of
! * loads by a factor of number of bytes per row. This improves performance
! * greatly, unfortunately it add more complexity to the code.
! *
! * Global participants in parsing logic:
! *
! * line_buf.cursor -- an offset from beginning of the line buffer
! * that indicates where we are about to begin the next scan. Note that
! * if we have WITH OIDS this cursor is already shifted past the first
! * OID attribute.
! *
! * attr_buf.cursor -- an offset from the beginning of the
! * attribute buffer that indicates where the current attribute begins.
! */
! static void
! CopyReadAttributesText(const char *delim, const char *escape, const char *null_print,
! int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr)
! {
! char delimc = delim[0]; /* delimiter character */
! char escapec = escape[0]; /* escape character */
! char *scan_start; /* pointer to line buffer for scan start. */
! char *scan_end; /* pointer to line buffer where char was found */
! int attr_pre_len; /* attr raw len, before processing escapes */
! int attr_post_len; /* current attr len after escaping */
! int m; /* attribute index being parsed */
! int bytes_remaining;/* num bytes remaining to be scanned in line buf */
! int chunk_start; /* offset to beginning of line chunk to load */
! int chunk_len; /* length of chunk of data to load to attr buf */
! int oct_val; /* byte value for octal escapes */
! int hex_val; /* byte value for hexadecimal escapes */
! char hexchar; /* char that appears after \x for hex values */
! int attnum; /* attribute number being parsed */
! ListCell *cur; /* cursor to attribute list used for this COPY */
! int attribute;
!
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /* cursor is now > 0 if we copy WITH OIDS */
! scan_start = line_buf.data + line_buf.cursor;
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! attr_pre_len = 0;
! attr_post_len = 0;
!
! /*
! * Scan through the line buffer to read all attributes data
! */
! while (line_buf.cursor < line_buf.len)
! {
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! if ((scan_end = scanTextAttr(scan_start, delimc, escapec, bytes_remaining))
! == NULL)
! {
! /* GOT TO END OF LINE BUFFER */
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* don't count eol char(s) in attr and chunk len calculation */
! if (eol_type == EOL_CRNL)
! {
! attr_pre_len += bytes_remaining - 2;
! chunk_len = line_buf.len - chunk_start - 2;
! }
! else
! {
! attr_pre_len += bytes_remaining - 1;
! chunk_len = line_buf.len - chunk_start - 1;
! }
!
! /* check if this is a NULL value or data value (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(line_buf.data + line_buf.len - attr_pre_len - 1, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! attr_offsets[m] = attr_buf.cursor;
!
!
! /* load the last chunk, the whole buffer in most cases */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! line_buf.cursor += attr_pre_len + 2; /* skip eol char and
! * '\0' to exit loop */
!
! if (lnext(cur) != NULL)
! {
! /*
! * For an empty data line, the previous COPY code will
! * fail it during the conversion stage. We can fail it here
! * already, but then we will fail the regression tests b/c
! * of a different error message. that's why we return so we
! * can get the same error message that regress expects. ahh...
! * this conditional is unnecessary and should be removed soon.
! */
! if (line_buf.len > 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m + 1]->attname))));
! else
! return;
! }
! }
! else
! /* FOUND A DELIMITER OR ESCAPE */
! {
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! if (*scan_end == delimc) /* found a delimiter */
! {
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* (we don't include the delimiter ch in length) */
! attr_pre_len += scan_end - scan_start;
! /* (we don't include the delimiter ch in length) */
! attr_post_len += scan_end - scan_start;
!
! /* check if this is a null print or data (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(scan_end - attr_pre_len, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* set the pointer to next attribute position */
! attr_offsets[m] = attr_buf.cursor;
!
! /*
! * update buffer cursors to our current location, +1 to skip
! * the delimc
! */
! line_buf.cursor = scan_end - line_buf.data + 1;
! attr_buf.cursor += attr_post_len + 1;
!
! /* prepare scan for next attr */
! scan_start = line_buf.data + line_buf.cursor;
! cur = lnext(cur);
! attr_pre_len = 0;
! attr_post_len = 0;
! }
! else
! /* found an escape character */
! {
! char nextc = *(scan_end + 1);
! char newc;
! int skip = 2;
!
! chunk_len = (scan_end - line_buf.data) - chunk_start + 1;
!
! /* load a chunk of data */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! switch (nextc)
{
case '0':
case '1':
***************
*** 2348,2445 ****
case '5':
case '6':
case '7':
! /* handle \013 */
! {
! int val;
! val = OCTVALUE(c);
! if (line_buf.cursor < line_buf.len)
! {
! c = line_buf.data[line_buf.cursor];
! if (ISOCTAL(c))
! {
! line_buf.cursor++;
! val = (val << 3) + OCTVALUE(c);
! if (line_buf.cursor < line_buf.len)
{
! c = line_buf.data[line_buf.cursor];
! if (ISOCTAL(c))
{
! line_buf.cursor++;
! val = (val << 3) + OCTVALUE(c);
}
}
! }
! }
! c = val & 0377;
! }
break;
case 'x':
/* Handle \x3F */
! if (line_buf.cursor < line_buf.len)
! {
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
! line_buf.cursor++;
! if (line_buf.cursor < line_buf.len)
! {
! hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! line_buf.cursor++;
! val = (val << 4) + GetDecimalFromHex(hexchar);
! }
! }
! c = val & 0xff;
}
}
break;
case 'b':
! c = '\b';
break;
case 'f':
! c = '\f';
break;
case 'n':
! c = '\n';
break;
case 'r':
! c = '\r';
break;
case 't':
! c = '\t';
break;
case 'v':
! c = '\v';
break;
/*
! * in all other cases, take the char after '\'
! * literally
*/
}
! }
! appendStringInfoCharMacro(&attribute_buf, c);
}
! /* check whether raw input matched null marker */
! input_len = end_cursor - start_cursor;
! if (input_len == strlen(null_print) &&
! strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
*isnull = true;
! else
! *isnull = false;
! return attribute_buf.data;
! }
/*
! * Read the value of a single attribute in CSV mode,
* performing de-escaping as needed. Escaping does not follow the normal
* PostgreSQL text mode, but instead "standard" (i.e. common) CSV usage.
*
--- 2773,2944 ----
case '5':
case '6':
case '7':
! oct_val = OCTVALUE(nextc);
! nextc = *(scan_end + 2);
! /*
! * (no need for out bad access check since line if
! * buffered)
! */
! if (ISOCTAL(nextc))
{
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! nextc = *(scan_end + 3);
! if (ISOCTAL(nextc))
{
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
}
}
! newc = oct_val & 0377; /* the escaped byte value */
break;
case 'x':
/* Handle \x3F */
! hexchar = *(scan_end + 2);
if (isxdigit(hexchar))
{
! skip++;
! hex_val = GetDecimalFromHex(hexchar);
! hexchar = *(scan_end + 3);
if (isxdigit(hexchar))
{
! skip++;
! hex_val = (hex_val << 4) + GetDecimalFromHex(hexchar);
}
+ newc = hex_val & 0xff;
}
+ else /* "\x" with no hex value */
+ newc = nextc;
break;
case 'b':
! newc = '\b';
break;
case 'f':
! newc = '\f';
break;
case 'n':
! newc = '\n';
break;
case 'r':
! newc = '\r';
break;
case 't':
! newc = '\t';
break;
case 'v':
! newc = '\v';
! break;
! default:
! if (nextc == delimc)
! newc = delimc;
! else if (nextc == escapec)
! newc = escapec;
! else
! /* no escape sequence, take next char literaly */
! newc = nextc;
break;
+ }
+
+ /* update to current length, add escape and escaped chars */
+ attr_pre_len += scan_end - scan_start + 2;
+ /* update to current length, escaped char */
+ attr_post_len += scan_end - scan_start + 1;
/*
! * Need to get rid of the escape character. This is done by
! * loading the chunk up to including the escape character
! * into the attribute buffer. Then overwritting the backslash
! * with the escaped sequence or char, and continuing to scan
! * from *after* the char than is after the escape in line buf.
*/
+ *(attr_buf.data + attr_buf.len - 1) = newc;
+ line_buf.cursor = scan_end - line_buf.data + skip;
+ scan_start = scan_end + skip;
+ chunk_start = line_buf.cursor;
+ chunk_len = 0;
}
!
! } /* end delimiter/backslash */
!
! } /* end line buffer scan. */
!
! /*
! * Replace all delimiters with NULL for string termination.
! * NOTE: only delimiters (NOT necessarily all delimc) are replaced.
! * Example (delimc = '|'):
! * - Before: f 1 | f \| 2 | f 3
! * - After : f 1 \0 f | 2 \0 f 3
! */
! for (attribute = 1; attribute < num_phys_attrs; attribute++)
! {
! if (attr_offsets[attribute] != 0)
! *(attr_buf.data + attr_offsets[attribute] - 1) = '\0';
}
! }
!
!
! /*
! * Read a binary attribute
! */
! static Datum
! CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo, Oid typioparam,
! bool *isnull)
! {
! int32 fld_size;
! Datum result;
!
! fld_size = CopyGetInt32();
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
! if (fld_size == -1)
! {
*isnull = true;
! return (Datum) 0;
! }
! if (fld_size < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid field size")));
! /* reset attr_buf to empty, and load raw data in it */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! enlargeStringInfo(&attr_buf, fld_size);
!
! CopyGetData(attr_buf.data, fld_size);
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
+ attr_buf.len = fld_size;
+ attr_buf.data[fld_size] = '\0';
+
+ /* Call the column type's binary input converter */
+ result = FunctionCall2(flinfo,
+ PointerGetDatum(&attr_buf),
+ ObjectIdGetDatum(typioparam));
+
+ /* Trouble if it didn't eat the whole buffer */
+ if (attr_buf.cursor != attr_buf.len)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
+ errmsg("incorrect binary data format")));
+
+ *isnull = false;
+ return result;
+ }
/*
! * Read all the attributes of the data line in CSV mode,
* performing de-escaping as needed. Escaping does not follow the normal
* PostgreSQL text mode, but instead "standard" (i.e. common) CSV usage.
*
***************
*** 2448,2472 ****
*
* null_print is the null marker string. Note that this is compared to
* the pre-de-escaped input string (thus if it is quoted it is not a NULL).
- *
- * *result is set to indicate what terminated the read:
- * NORMAL_ATTR: column delimiter
- * END_OF_LINE: end of line
- * UNTERMINATED_FIELD no quote detected at end of a quoted field
- *
- * In any case, the string read up to the terminator (or end of file)
- * is returned.
- *
- * *isnull is set true or false depending on whether the input matched
- * the null marker. Note that the caller cannot check this since the
- * returned string will be the post-de-escaping equivalent, which may
- * look the same as some valid data string.
*----------
*/
!
! static char *
! CopyReadAttributeCSV(const char *delim, const char *null_print, char *quote,
! char *escape, CopyReadResult *result, bool *isnull)
{
char delimc = delim[0];
char quotec = quote[0];
--- 2947,2958 ----
*
* null_print is the null marker string. Note that this is compared to
* the pre-de-escaped input string (thus if it is quoted it is not a NULL).
*----------
*/
! static void
! CopyReadAttributesCSV(const char *delim, const char *null_print, char *quote,
! char *escape, int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr)
{
char delimc = delim[0];
char quotec = quote[0];
***************
*** 2477,2502 ****
int input_len;
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
- /* set default status */
- *result = END_OF_LINE;
for (;;)
{
end_cursor = line_buf.cursor;
if (line_buf.cursor >= line_buf.len)
break;
c = line_buf.data[line_buf.cursor++];
/* unquoted field delimiter */
if (!in_quote && c == delimc)
{
! *result = NORMAL_ATTR;
! break;
}
/* start of quoted field (or part of field) */
--- 2963,3043 ----
int input_len;
bool in_quote = false;
bool saw_quote = false;
+ int attnum; /* attribute number being parsed */
+ int m; /* attribute index being parsed */
+ ListCell *cur; /* cursor to attribute list used for this COPY */
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! input_len = 0;
for (;;)
{
end_cursor = line_buf.cursor;
+
+ /* finished processing attributes in line */
if (line_buf.cursor >= line_buf.len)
+ {
+ /* check whether raw input matched null marker */
+ input_len = end_cursor - start_cursor;
+ if (!saw_quote && input_len == null_print_len &&
+ strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
+ nulls[m] = 'n';
+ else
+ nulls[m] = ' ';
+
+ if (in_quote)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("unterminated CSV quoted field")));
+
+ if (lnext(cur) != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(attr[m + 1]->attname))));
break;
+ }
+
c = line_buf.data[line_buf.cursor++];
/* unquoted field delimiter */
if (!in_quote && c == delimc)
{
! /* check whether raw input matched null marker */
! input_len = end_cursor - start_cursor;
! if (!saw_quote && input_len == null_print_len &&
! strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* terminate attr string with '\0' */
! appendStringInfoCharMacro(&attr_buf, '\0');
! attr_buf.cursor++;
!
! /* setup next attribute scan */
! cur = lnext(cur);
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
! attr_offsets[m] = attr_buf.cursor;
! start_cursor = line_buf.cursor;
! continue;
}
/* start of quoted field (or part of field) */
***************
*** 2520,2527 ****
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attribute_buf, nextc);
line_buf.cursor++;
continue;
}
}
--- 3061,3069 ----
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attr_buf, nextc);
line_buf.cursor++;
+ attr_buf.cursor++;
continue;
}
}
***************
*** 2537,2616 ****
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
- if (in_quote)
- *result = UNTERMINATED_FIELD;
-
- /* check whether raw input matched null marker */
- input_len = end_cursor - start_cursor;
- if (!saw_quote && input_len == strlen(null_print) &&
- strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
- *isnull = true;
- else
- *isnull = false;
-
- return attribute_buf.data;
}
/*
! * Read a binary attribute
*/
! static Datum
! CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, int32 typmod,
bool *isnull)
{
! int32 fld_size;
! Datum result;
!
! fld_size = CopyGetInt32();
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
! if (fld_size == -1)
! {
! *isnull = true;
! return (Datum) 0;
}
! if (fld_size < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! attribute_buf.cursor = 0;
! enlargeStringInfo(&attribute_buf, fld_size);
- CopyGetData(attribute_buf.data, fld_size);
- if (CopyGetEof())
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("unexpected EOF in COPY data")));
! attribute_buf.len = fld_size;
! attribute_buf.data[fld_size] = '\0';
! /* Call the column type's binary input converter */
! result = FunctionCall3(flinfo,
! PointerGetDatum(&attribute_buf),
! ObjectIdGetDatum(typioparam),
! Int32GetDatum(typmod));
- /* Trouble if it didn't eat the whole buffer */
- if (attribute_buf.cursor != attribute_buf.len)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
- errmsg("incorrect binary data format")));
- *isnull = false;
- return result;
- }
/*
* Send text representation of one attribute, with conversion and escaping
--- 3079,3145 ----
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attr_buf, c);
! attr_buf.cursor++;
}
}
/*
! * Read the first attribute. This is mainly used to maintain support
! * for an OID column. All the rest of the columns will be read at once with
! * CopyReadAttributesText.
*/
! static char *
! CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
bool *isnull)
{
! char delimc = delim[0];
! char *start_loc = line_buf.data + line_buf.cursor;
! char *end_loc;
! int attr_len = 0;
! int bytes_remaining;
!
! /* reset attribute buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! /* # of bytes that were not yet processed in this line */
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! /* got to end of line */
! if ((end_loc = scanTextLine(start_loc, delimc, bytes_remaining)) == NULL)
! {
! attr_len = bytes_remaining - 1; /* don't count '\n' in len calculation */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 2; /* skip '\n' and '\0' */
}
! else
! /* found a delimiter */
! {
! /*
! * (we don't care if delim was preceded with a backslash, because it's
! * an invalid OID anyway)
! */
! attr_len = end_loc - start_loc; /* we don't include the delimiter ch */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 1;
! }
! /* check whether raw input matched null marker */
! if (attr_len == null_print_len && strncmp(start_loc, null_print, attr_len) == 0)
! *isnull = true;
! else
! *isnull = false;
! return attr_buf.data;
! }
/*
* Send text representation of one attribute, with conversion and escaping
***************
*** 2800,2802 ****
--- 3329,3432 ----
return attnums;
}
+
+ /*
+ * The following are custom versions of the string function strchr().
+ * As opposed to the original strchr which searches through
+ * a string until the target character is found, or a NULL is
+ * found, this version will not return when a NULL is found.
+ * Instead it will search through a pre-defined length of
+ * bytes and will return only if the target character(s) is reached.
+ *
+ * If our client encoding is not a supported server encoding, we
+ * know that it is not safe to look at each character as trailing
+ * byte in a multibyte character may be a 7-bit ASCII equivalent.
+ * Therefore we use pg_encoding_mblen to skip to the end of the
+ * character.
+ *
+ * returns:
+ * pointer to c - if c is located within the string.
+ * NULL - if c was not found in specified length of search. Note:
+ * this DOESN'T mean that a '\0' was reached.
+ */
+ char *
+ scanTextLine(const char *s, char eol, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only && !line_buf_converted)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ for (start = s; *s != eol && s < start + len; s += mblen)
+ mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ return ((*s == eol) ? (char *) s : NULL);
+ }
+ else
+ return memchr(s,eol,len);
+ }
+
+
+ char *
+ scanCSVLine(const char *s, char eol, char escapec, char quotec, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only && !line_buf_converted)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ for (start = s; *s != eol && s < start + len; s += mblen)
+ {
+ if (in_quote && *s == escapec)
+ last_was_esc = !last_was_esc;
+ if (*s == quotec && !last_was_esc)
+ in_quote = !in_quote;
+ if (*s != escapec)
+ last_was_esc = false;
+
+ mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+ }
+ }
+ else
+ /* safe to scroll byte by byte */
+ {
+ for (start = s; *s != eol && s < start + len; s++)
+ {
+ if (in_quote && *s == escapec)
+ last_was_esc = !last_was_esc;
+ if (*s == quotec && !last_was_esc)
+ in_quote = !in_quote;
+ if (*s != escapec)
+ last_was_esc = false;
+ }
+
+ }
+
+ if(*s != escapec) /* found eol_ch */
+ last_was_esc = false;
+
+ return ((*s == eol) ? (char *) s : NULL);
+ }
+
+ /*
+ * Scan for 1 of 2 characters necessary for attribute parsing. No need
+ * for checking for multi-byte characters since conversion already
+ * happened.
+ */
+ char *
+ scanTextAttr(const char *s, char c1, char c2, size_t len)
+ {
+ const char *start;
+
+ for (start = s; *s != c1 && *s != c2 && s < start + len; s++)
+ ;
+
+ return (*s != '\0' ? (char *) s : NULL);
+ }
+
+
+
+
+
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-07-19 15:23 ` Andrew Dunstan <andrew@dunslane.net>
2 siblings, 0 replies; 52+ messages in thread
From: Andrew Dunstan @ 2005-07-19 15:23 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Alon Goldshuv wrote:
>I revisited my patch and removed the code duplications that were there, and
>added support for CSV with buffered input, so CSV now runs faster too
>(although it is not as optimized as the TEXT format parsing). So now
>TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
>
>Patch attached.
>
>
>
I do not have time to review this 2900 line patch analytically, nor to
benchmark it. I have done some functional testing of it on Windows, and
tried to break it in text and CSV modes, and with both Unix and Windows
type line endings - I have not observed any breakage.
This does need lots of eyeballs, though.
cheers
andrew
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-07-19 19:54 ` Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2 siblings, 1 reply; 52+ messages in thread
From: Mark Wong @ 2005-07-19 19:54 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
On Thu, 14 Jul 2005 17:22:18 -0700
"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
> I revisited my patch and removed the code duplications that were there, and
> added support for CSV with buffered input, so CSV now runs faster too
> (although it is not as optimized as the TEXT format parsing). So now
> TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
Hi Alon,
I'm curious, what kind of system are you testing this on? I'm trying to
load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
interested in the results you would expect.
Mark
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-19 21:05 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
0 siblings, 1 reply; 52+ messages in thread
From: Alon Goldshuv @ 2005-07-19 21:05 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: pgsql-patches@postgresql.org
Hi Mark,
I improved the data *parsing* capabilities of COPY, and didn't touch the
data conversion or data insertion parts of the code. The parsing improvement
will vary largely depending on the ratio of parsing -to- converting and
inserting.
Therefore, the speed increase really depends on the nature of your data:
100GB file with
long data rows (lots of parsing)
Small number of columns (small number of attr conversions per row)
less rows (less tuple insertions)
Will show the best performance improvements.
However, same file size 100GB with
Short data rows (minimal parsing)
large number of columns (large number of attr conversions per row)
AND/OR
more rows (more tuple insertions)
Will show improvements but not as significant.
In general I'll estimate 40%-95% improvement in load speed for the 1st case
and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
etc... This is for TEXT format. As for CSV, it may be faster but not as much
as I specified here. BINARY will stay the same as before.
HTH
Alon.
On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
> On Thu, 14 Jul 2005 17:22:18 -0700
> "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>
>> I revisited my patch and removed the code duplications that were there, and
>> added support for CSV with buffered input, so CSV now runs faster too
>> (although it is not as optimized as the TEXT format parsing). So now
>> TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
>
> Hi Alon,
>
> I'm curious, what kind of system are you testing this on? I'm trying to
> load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
> interested in the results you would expect.
>
> Mark
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-07-19 21:37 ` Mark Wong <markw@osdl.org>
2005-07-19 22:06 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 22:17 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
0 siblings, 2 replies; 52+ messages in thread
From: Mark Wong @ 2005-07-19 21:37 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Hi Alon,
Yeah, that helps. I just need to break up my scripts a little to just
load the data and not build indexes.
Is the following information good enough to give a guess about the data
I'm loading, if you don't mind? ;) Here's a link to my script to create
tables:
http://developer.osdl.org/markw/mt/getfile.py?id=eaf16b7831588729780645b2bb44f7f23437e432&path=s...
File sizes:
-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 customer.tbl
-rw-r--r-- 1 markw 50 74G Jul 8 15:03 lineitem.tbl
-rw-r--r-- 1 markw 50 2.1K Jul 8 15:03 nation.tbl
-rw-r--r-- 1 markw 50 17G Jul 8 15:03 orders.tbl
-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 part.tbl
-rw-r--r-- 1 markw 50 12G Jul 8 15:03 partsupp.tbl
-rw-r--r-- 1 markw 50 391 Jul 8 15:03 region.tbl
-rw-r--r-- 1 markw 50 136M Jul 8 15:03 supplier.tbl
Number of rows:
# wc -l *.tbl
15000000 customer.tbl
600037902 lineitem.tbl
25 nation.tbl
150000000 orders.tbl
20000000 part.tbl
80000000 partsupp.tbl
5 region.tbl
1000000 supplier.tbl
Thanks,
Mark
On Tue, 19 Jul 2005 14:05:56 -0700
"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
> Hi Mark,
>
> I improved the data *parsing* capabilities of COPY, and didn't touch the
> data conversion or data insertion parts of the code. The parsing improvement
> will vary largely depending on the ratio of parsing -to- converting and
> inserting.
>
> Therefore, the speed increase really depends on the nature of your data:
>
> 100GB file with
> long data rows (lots of parsing)
> Small number of columns (small number of attr conversions per row)
> less rows (less tuple insertions)
>
> Will show the best performance improvements.
>
> However, same file size 100GB with
> Short data rows (minimal parsing)
> large number of columns (large number of attr conversions per row)
> AND/OR
> more rows (more tuple insertions)
>
> Will show improvements but not as significant.
> In general I'll estimate 40%-95% improvement in load speed for the 1st case
> and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
> etc... This is for TEXT format. As for CSV, it may be faster but not as much
> as I specified here. BINARY will stay the same as before.
>
> HTH
> Alon.
>
>
>
>
>
>
> On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
>
> > On Thu, 14 Jul 2005 17:22:18 -0700
> > "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
> >
> >> I revisited my patch and removed the code duplications that were there, and
> >> added support for CSV with buffered input, so CSV now runs faster too
> >> (although it is not as optimized as the TEXT format parsing). So now
> >> TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
> >
> > Hi Alon,
> >
> > I'm curious, what kind of system are you testing this on? I'm trying to
> > load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
> > interested in the results you would expect.
> >
> > Mark
> >
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-19 22:06 ` Alon Goldshuv <agoldshuv@greenplum.com>
1 sibling, 0 replies; 52+ messages in thread
From: Alon Goldshuv @ 2005-07-19 22:06 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: pgsql-patches@postgresql.org
Mark,
Thanks for the info.
Yes, isolating indexes out of the picture is a good idea for this purpose.
I can't really give a guess to how fast the load rate should be. I don't
know how your system is configured, and all the hardware characteristics
(and even if I knew that info I may not be able to guess...). I am pretty
confident that the load will be faster than before, I'll risk that ;-)
Looking into your TPC-H size and metadata I'll estimate that
partsupp,customer and orders will have the most significant increase in load
rate. You could start with those.
I guess the only way to really know is to try... Load several times with the
existing PG-COPY and then load several times with the patched COPY and
compare. I'll be curious to hear your results.
Thx,
Alon.
On 7/19/05 2:37 PM, "Mark Wong" <markw@osdl.org> wrote:
> Hi Alon,
>
> Yeah, that helps. I just need to break up my scripts a little to just
> load the data and not build indexes.
>
> Is the following information good enough to give a guess about the data
> I'm loading, if you don't mind? ;) Here's a link to my script to create
> tables:
> http://developer.osdl.org/markw/mt/getfile.py?id=eaf16b7831588729780645b2bb44f
> 7f23437e432&path=scripts/pgsql/create_tables.sh.in
>
> File sizes:
> -rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 customer.tbl
> -rw-r--r-- 1 markw 50 74G Jul 8 15:03 lineitem.tbl
> -rw-r--r-- 1 markw 50 2.1K Jul 8 15:03 nation.tbl
> -rw-r--r-- 1 markw 50 17G Jul 8 15:03 orders.tbl
> -rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 part.tbl
> -rw-r--r-- 1 markw 50 12G Jul 8 15:03 partsupp.tbl
> -rw-r--r-- 1 markw 50 391 Jul 8 15:03 region.tbl
> -rw-r--r-- 1 markw 50 136M Jul 8 15:03 supplier.tbl
>
> Number of rows:
> # wc -l *.tbl
> 15000000 customer.tbl
> 600037902 lineitem.tbl
> 25 nation.tbl
> 150000000 orders.tbl
> 20000000 part.tbl
> 80000000 partsupp.tbl
> 5 region.tbl
> 1000000 supplier.tbl
>
> Thanks,
> Mark
>
> On Tue, 19 Jul 2005 14:05:56 -0700
> "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>
>> Hi Mark,
>>
>> I improved the data *parsing* capabilities of COPY, and didn't touch the
>> data conversion or data insertion parts of the code. The parsing improvement
>> will vary largely depending on the ratio of parsing -to- converting and
>> inserting.
>>
>> Therefore, the speed increase really depends on the nature of your data:
>>
>> 100GB file with
>> long data rows (lots of parsing)
>> Small number of columns (small number of attr conversions per row)
>> less rows (less tuple insertions)
>>
>> Will show the best performance improvements.
>>
>> However, same file size 100GB with
>> Short data rows (minimal parsing)
>> large number of columns (large number of attr conversions per row)
>> AND/OR
>> more rows (more tuple insertions)
>>
>> Will show improvements but not as significant.
>> In general I'll estimate 40%-95% improvement in load speed for the 1st case
>> and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
>> etc... This is for TEXT format. As for CSV, it may be faster but not as much
>> as I specified here. BINARY will stay the same as before.
>>
>> HTH
>> Alon.
>>
>>
>>
>>
>>
>>
>> On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
>>
>>> On Thu, 14 Jul 2005 17:22:18 -0700
>>> "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>>>
>>>> I revisited my patch and removed the code duplications that were there, and
>>>> added support for CSV with buffered input, so CSV now runs faster too
>>>> (although it is not as optimized as the TEXT format parsing). So now
>>>> TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original
>>>> file.
>>>
>>> Hi Alon,
>>>
>>> I'm curious, what kind of system are you testing this on? I'm trying to
>>> load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
>>> interested in the results you would expect.
>>>
>>> Mark
>>>
>>
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-19 22:17 ` Andrew Dunstan <andrew@dunslane.net>
2005-07-19 22:51 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
1 sibling, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2005-07-19 22:17 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Mark,
You should definitely not be doing this sort of thing, I believe:
CREATE TABLE orders (
o_orderkey INTEGER,
o_custkey INTEGER,
o_orderstatus CHAR(1),
o_totalprice REAL,
o_orderDATE DATE,
o_orderpriority CHAR(15),
o_clerk CHAR(15),
o_shippriority INTEGER,
o_comment VARCHAR(79),
PRIMARY KEY (o_orderkey))
Create the table with no constraints, load the data, then set up primary keys and whatever other constraints you want using ALTER TABLE. Last time I did a load like this (albeit 2 orders of magnitude smaller) I saw a 50% speedup from deferring constarint creation.
cheers
andrew
Mark Wong wrote:
>Hi Alon,
>
>Yeah, that helps. I just need to break up my scripts a little to just
>load the data and not build indexes.
>
>Is the following information good enough to give a guess about the data
>I'm loading, if you don't mind? ;) Here's a link to my script to create
>tables:
>http://developer.osdl.org/markw/mt/getfile.py?id=eaf16b7831588729780645b2bb44f7f23437e432&path=s...
>
>File sizes:
>-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 customer.tbl
>-rw-r--r-- 1 markw 50 74G Jul 8 15:03 lineitem.tbl
>-rw-r--r-- 1 markw 50 2.1K Jul 8 15:03 nation.tbl
>-rw-r--r-- 1 markw 50 17G Jul 8 15:03 orders.tbl
>-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 part.tbl
>-rw-r--r-- 1 markw 50 12G Jul 8 15:03 partsupp.tbl
>-rw-r--r-- 1 markw 50 391 Jul 8 15:03 region.tbl
>-rw-r--r-- 1 markw 50 136M Jul 8 15:03 supplier.tbl
>
>Number of rows:
># wc -l *.tbl
> 15000000 customer.tbl
> 600037902 lineitem.tbl
> 25 nation.tbl
> 150000000 orders.tbl
> 20000000 part.tbl
> 80000000 partsupp.tbl
> 5 region.tbl
> 1000000 supplier.tbl
>
>Thanks,
>Mark
>
>On Tue, 19 Jul 2005 14:05:56 -0700
>"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>
>
>
>>Hi Mark,
>>
>>I improved the data *parsing* capabilities of COPY, and didn't touch the
>>data conversion or data insertion parts of the code. The parsing improvement
>>will vary largely depending on the ratio of parsing -to- converting and
>>inserting.
>>
>>Therefore, the speed increase really depends on the nature of your data:
>>
>>100GB file with
>>long data rows (lots of parsing)
>>Small number of columns (small number of attr conversions per row)
>>less rows (less tuple insertions)
>>
>>Will show the best performance improvements.
>>
>>However, same file size 100GB with
>>Short data rows (minimal parsing)
>>large number of columns (large number of attr conversions per row)
>>AND/OR
>>more rows (more tuple insertions)
>>
>>Will show improvements but not as significant.
>>In general I'll estimate 40%-95% improvement in load speed for the 1st case
>>and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
>>etc... This is for TEXT format. As for CSV, it may be faster but not as much
>>as I specified here. BINARY will stay the same as before.
>>
>>HTH
>>Alon.
>>
>>
>>
>>
>>
>>
>>On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
>>
>>
>>
>>>On Thu, 14 Jul 2005 17:22:18 -0700
>>>"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>>>
>>>
>>>
>>>>I revisited my patch and removed the code duplications that were there, and
>>>>added support for CSV with buffered input, so CSV now runs faster too
>>>>(although it is not as optimized as the TEXT format parsing). So now
>>>>TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
>>>>
>>>>
>>>Hi Alon,
>>>
>>>I'm curious, what kind of system are you testing this on? I'm trying to
>>>load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
>>>interested in the results you would expect.
>>>
>>>Mark
>>>
>>>
>>>
>
>---------------------------(end of broadcast)---------------------------
>TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>
>
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 22:17 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
@ 2005-07-19 22:51 ` Mark Wong <markw@osdl.org>
2005-07-20 00:39 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Mark Wong @ 2005-07-19 22:51 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Whoopsies, yeah good point about the PRIMARY KEY. I'll fix that.
Mark
On Tue, 19 Jul 2005 18:17:52 -0400
Andrew Dunstan <andrew@dunslane.net> wrote:
> Mark,
>
> You should definitely not be doing this sort of thing, I believe:
>
> CREATE TABLE orders (
> o_orderkey INTEGER,
> o_custkey INTEGER,
> o_orderstatus CHAR(1),
> o_totalprice REAL,
> o_orderDATE DATE,
> o_orderpriority CHAR(15),
> o_clerk CHAR(15),
> o_shippriority INTEGER,
> o_comment VARCHAR(79),
> PRIMARY KEY (o_orderkey))
>
> Create the table with no constraints, load the data, then set up primary keys and whatever other constraints you want using ALTER TABLE. Last time I did a load like this (albeit 2 orders of magnitude smaller) I saw a 50% speedup from deferring constarint creation.
>
>
> cheers
>
> andrew
>
>
>
> Mark Wong wrote:
>
> >Hi Alon,
> >
> >Yeah, that helps. I just need to break up my scripts a little to just
> >load the data and not build indexes.
> >
> >Is the following information good enough to give a guess about the data
> >I'm loading, if you don't mind? ;) Here's a link to my script to create
> >tables:
> >http://developer.osdl.org/markw/mt/getfile.py?id=eaf16b7831588729780645b2bb44f7f23437e432&path=s...
> >
> >File sizes:
> >-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 customer.tbl
> >-rw-r--r-- 1 markw 50 74G Jul 8 15:03 lineitem.tbl
> >-rw-r--r-- 1 markw 50 2.1K Jul 8 15:03 nation.tbl
> >-rw-r--r-- 1 markw 50 17G Jul 8 15:03 orders.tbl
> >-rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 part.tbl
> >-rw-r--r-- 1 markw 50 12G Jul 8 15:03 partsupp.tbl
> >-rw-r--r-- 1 markw 50 391 Jul 8 15:03 region.tbl
> >-rw-r--r-- 1 markw 50 136M Jul 8 15:03 supplier.tbl
> >
> >Number of rows:
> ># wc -l *.tbl
> > 15000000 customer.tbl
> > 600037902 lineitem.tbl
> > 25 nation.tbl
> > 150000000 orders.tbl
> > 20000000 part.tbl
> > 80000000 partsupp.tbl
> > 5 region.tbl
> > 1000000 supplier.tbl
> >
> >Thanks,
> >Mark
> >
> >On Tue, 19 Jul 2005 14:05:56 -0700
> >"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
> >
> >
> >
> >>Hi Mark,
> >>
> >>I improved the data *parsing* capabilities of COPY, and didn't touch the
> >>data conversion or data insertion parts of the code. The parsing improvement
> >>will vary largely depending on the ratio of parsing -to- converting and
> >>inserting.
> >>
> >>Therefore, the speed increase really depends on the nature of your data:
> >>
> >>100GB file with
> >>long data rows (lots of parsing)
> >>Small number of columns (small number of attr conversions per row)
> >>less rows (less tuple insertions)
> >>
> >>Will show the best performance improvements.
> >>
> >>However, same file size 100GB with
> >>Short data rows (minimal parsing)
> >>large number of columns (large number of attr conversions per row)
> >>AND/OR
> >>more rows (more tuple insertions)
> >>
> >>Will show improvements but not as significant.
> >>In general I'll estimate 40%-95% improvement in load speed for the 1st case
> >>and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
> >>etc... This is for TEXT format. As for CSV, it may be faster but not as much
> >>as I specified here. BINARY will stay the same as before.
> >>
> >>HTH
> >>Alon.
> >>
> >>
> >>
> >>
> >>
> >>
> >>On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
> >>
> >>
> >>
> >>>On Thu, 14 Jul 2005 17:22:18 -0700
> >>>"Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
> >>>
> >>>
> >>>
> >>>>I revisited my patch and removed the code duplications that were there, and
> >>>>added support for CSV with buffered input, so CSV now runs faster too
> >>>>(although it is not as optimized as the TEXT format parsing). So now
> >>>>TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original file.
> >>>>
> >>>>
> >>>Hi Alon,
> >>>
> >>>I'm curious, what kind of system are you testing this on? I'm trying to
> >>>load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
> >>>interested in the results you would expect.
> >>>
> >>>Mark
> >>>
> >>>
> >>>
> >
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 19:54 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-19 22:17 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-07-19 22:51 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-20 00:39 ` Luke Lonergan <llonergan@greenplum.com>
0 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-07-20 00:39 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Good points on all, another element in the performance expectations is the
ratio of CPU speed to I/O subsystem speed, as Alon had hinted earlier.
This patch substantially (500%) improves the efficiency of parsing in the
COPY path, which, on a 3GHz P4 desktop with a commodity disk drive
represents 8 of a total of 30 seconds of processing time. So, by reducing
the parsing time from 8 seconds to 1.5 seconds, the overall COPY time is
reduced from 30 seconds to 23.5 seconds, or a speedup of about 20%.
On a dual 2.2GHz Opteron machine with a 6-disk SCSI RAID subsystem capable
of 240MB/s sequential read and writes, the ratios change and we see between
35% and 95% increase in COPY performance, with the bottleneck being CPU.
The disk is only running at about 90MB/s during this period.
I'd expect that as your CPUs slow down relative to your I/O speed, and
Itaniums or IT2s are quite slow, you should see an increased effect of the
parsing improvements.
One good way to validate the effect is to watch the I/O bandwidth using
vmstat 1 (on Linux) while the load is progressing. When you watch that with
the unpatched source and with the patched source, if they are the same, you
should see no benefit from the patch (you are I/O limited).
If you check your underlying sequential write speed, you will be
bottlenecked at roughly half that in performing COPY because of the
write-through the WAL.
- Luke
On 7/19/05 3:51 PM, "Mark Wong" <markw@osdl.org> wrote:
> Whoopsies, yeah good point about the PRIMARY KEY. I'll fix that.
>
> Mark
>
> On Tue, 19 Jul 2005 18:17:52 -0400
> Andrew Dunstan <andrew@dunslane.net> wrote:
>
>> Mark,
>>
>> You should definitely not be doing this sort of thing, I believe:
>>
>> CREATE TABLE orders (
>> o_orderkey INTEGER,
>> o_custkey INTEGER,
>> o_orderstatus CHAR(1),
>> o_totalprice REAL,
>> o_orderDATE DATE,
>> o_orderpriority CHAR(15),
>> o_clerk CHAR(15),
>> o_shippriority INTEGER,
>> o_comment VARCHAR(79),
>> PRIMARY KEY (o_orderkey))
>>
>> Create the table with no constraints, load the data, then set up primary keys
>> and whatever other constraints you want using ALTER TABLE. Last time I did a
>> load like this (albeit 2 orders of magnitude smaller) I saw a 50% speedup
>> from deferring constarint creation.
>>
>>
>> cheers
>>
>> andrew
>>
>>
>>
>> Mark Wong wrote:
>>
>>> Hi Alon,
>>>
>>> Yeah, that helps. I just need to break up my scripts a little to just
>>> load the data and not build indexes.
>>>
>>> Is the following information good enough to give a guess about the data
>>> I'm loading, if you don't mind? ;) Here's a link to my script to create
>>> tables:
>>> http://developer.osdl.org/markw/mt/getfile.py?id=eaf16b7831588729780645b2bb4
>>> 4f7f23437e432&path=scripts/pgsql/create_tables.sh.in
>>>
>>> File sizes:
>>> -rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 customer.tbl
>>> -rw-r--r-- 1 markw 50 74G Jul 8 15:03 lineitem.tbl
>>> -rw-r--r-- 1 markw 50 2.1K Jul 8 15:03 nation.tbl
>>> -rw-r--r-- 1 markw 50 17G Jul 8 15:03 orders.tbl
>>> -rw-r--r-- 1 markw 50 2.3G Jul 8 15:03 part.tbl
>>> -rw-r--r-- 1 markw 50 12G Jul 8 15:03 partsupp.tbl
>>> -rw-r--r-- 1 markw 50 391 Jul 8 15:03 region.tbl
>>> -rw-r--r-- 1 markw 50 136M Jul 8 15:03 supplier.tbl
>>>
>>> Number of rows:
>>> # wc -l *.tbl
>>> 15000000 customer.tbl
>>> 600037902 lineitem.tbl
>>> 25 nation.tbl
>>> 150000000 orders.tbl
>>> 20000000 part.tbl
>>> 80000000 partsupp.tbl
>>> 5 region.tbl
>>> 1000000 supplier.tbl
>>>
>>> Thanks,
>>> Mark
>>>
>>> On Tue, 19 Jul 2005 14:05:56 -0700
>>> "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>>>
>>>
>>>
>>>> Hi Mark,
>>>>
>>>> I improved the data *parsing* capabilities of COPY, and didn't touch the
>>>> data conversion or data insertion parts of the code. The parsing
>>>> improvement
>>>> will vary largely depending on the ratio of parsing -to- converting and
>>>> inserting.
>>>>
>>>> Therefore, the speed increase really depends on the nature of your data:
>>>>
>>>> 100GB file with
>>>> long data rows (lots of parsing)
>>>> Small number of columns (small number of attr conversions per row)
>>>> less rows (less tuple insertions)
>>>>
>>>> Will show the best performance improvements.
>>>>
>>>> However, same file size 100GB with
>>>> Short data rows (minimal parsing)
>>>> large number of columns (large number of attr conversions per row)
>>>> AND/OR
>>>> more rows (more tuple insertions)
>>>>
>>>> Will show improvements but not as significant.
>>>> In general I'll estimate 40%-95% improvement in load speed for the 1st case
>>>> and 10%-40% for the 2nd. But that also depends on the hardware, disk speed
>>>> etc... This is for TEXT format. As for CSV, it may be faster but not as
>>>> much
>>>> as I specified here. BINARY will stay the same as before.
>>>>
>>>> HTH
>>>> Alon.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 7/19/05 12:54 PM, "Mark Wong" <markw@osdl.org> wrote:
>>>>
>>>>
>>>>
>>>>> On Thu, 14 Jul 2005 17:22:18 -0700
>>>>> "Alon Goldshuv" <agoldshuv@greenplum.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> I revisited my patch and removed the code duplications that were there,
>>>>>> and
>>>>>> added support for CSV with buffered input, so CSV now runs faster too
>>>>>> (although it is not as optimized as the TEXT format parsing). So now
>>>>>> TEXT,CSV and BINARY are all parsed in CopyFrom(), like in the original
>>>>>> file.
>>>>>>
>>>>>>
>>>>> Hi Alon,
>>>>>
>>>>> I'm curious, what kind of system are you testing this on? I'm trying to
>>>>> load 100GB of data in our dbt3 workload on a 4-way itanium2. I'm
>>>>> interested in the results you would expect.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-08-02 14:59 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 52+ messages in thread
From: Alon Goldshuv @ 2005-08-02 14:59 UTC (permalink / raw)
To: pgsql-patches@postgresql.org
New patch attached. It includes very minor changes. These are changes that
were committed to CVS 3 weeks ago (copy.c 1.247) which I missed in the
previous patch.
Alon.
Attachments:
copy_parse_improvements_V16.patch (application/octet-stream, 77.0K) download
Index: src/backend/commands/copy.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/copy.c,v
retrieving revision 1.247
diff -c -r1.247 copy.c
*** src/backend/commands/copy.c 10 Jul 2005 21:13:58 -0000 1.247
--- src/backend/commands/copy.c 2 Aug 2005 14:38:37 -0000
***************
*** 50,55 ****
--- 50,56 ----
#define ISOCTAL(c) (((c) >= '0') && ((c) <= '7'))
#define OCTVALUE(c) ((c) - '0')
+ #define COPY_BUF_SIZE 65536
/*
* Represents the different source/dest cases we need to worry about at
***************
*** 63,78 ****
} CopyDest;
/*
- * State indicator showing what stopped CopyReadAttribute()
- */
- typedef enum CopyReadResult
- {
- NORMAL_ATTR,
- END_OF_LINE,
- UNTERMINATED_FIELD
- } CopyReadResult;
-
- /*
* Represents the end-of-line terminator type of the input
*/
typedef enum EolType
--- 64,69 ----
***************
*** 97,102 ****
--- 88,96 ----
static EolType eol_type; /* EOL type of input */
static int client_encoding; /* remote side's character encoding */
static int server_encoding; /* local encoding */
+ static char eol_ch[2]; /* The byte values of the 1 or 2 eol bytes */
+ static bool client_encoding_only; /* true if client encoding is a non
+ * supported server encoding */
/* these are just for error messages, see copy_in_error_callback */
static bool copy_binary; /* is it a binary copy? */
***************
*** 104,126 ****
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attribute_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attribute_buf's data buffer!
*/
! static StringInfoData attribute_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attribute_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
--- 98,141 ----
static int copy_lineno; /* line number for error messages */
static const char *copy_attname; /* current att for error messages */
+ /*
+ * Static variables for buffered input parsing
+ */
+ static char input_buf[COPY_BUF_SIZE + 1]; /* extra byte for '\0' */
+ static int buffer_index; /* input buffer index */
+ static bool end_marker;
+ static char *begloc;
+ static char *endloc;
+ static bool buf_done; /* finished processing the current buffer */
+ static bool line_done; /* finished processing the whole line or
+ * stopped in the middle */
+ /* these are for CSV format */
+ static bool in_quote;
+ static bool last_was_esc;
+
+ /* these are for TEXT format */
+ static bool esc_in_prevbuf; /* escape was last character of the data
+ * input buffer */
+ static bool cr_in_prevbuf; /* CR was last character of the data input
+ * buffer */
+
/*
* These static variables are used to avoid incurring overhead for each
! * attribute processed. attr_buf is reused on each CopyReadAttribute
* call to hold the string being read in. Under normal use it will soon
* grow to a suitable size, and then we will avoid palloc/pfree overhead
* for subsequent attributes. Note that CopyReadAttribute returns a pointer
! * to attr_buf's data buffer!
*/
! static StringInfoData attr_buf;
/*
* Similarly, line_buf holds the whole input line being processed (its
* cursor field points to the next character to be read by CopyReadAttribute).
* The input cycle is first to read the whole line into line_buf, convert it
* to server encoding, and then extract individual attribute fields into
! * attr_buf. (We used to have CopyReadAttribute read the input source
* directly, but that caused a lot of encoding issues and unnecessary logic
* complexity.)
*/
***************
*** 137,155 ****
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLine(char * quote, char * escape);
! static char *CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull);
! static char *CopyReadAttributeCSV(const char *delim, const char *null_print,
! char *quote, char *escape,
! CopyReadResult *result, bool *isnull);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
Oid typioparam, int32 typmod, bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
--- 152,176 ----
static void CopyFrom(Relation rel, List *attnumlist, bool binary, bool oids,
char *delim, char *null_print, bool csv_mode, char *quote, char *escape,
List *force_notnull_atts, bool header_line);
! static bool CopyReadLineText(size_t bytesread, char *escape);
! static bool CopyReadLineCSV(size_t bytesread, char *quote, char *escape);
! static void CopyReadAttributesText(const char *delim, const char *escape, const char *null_print,
! int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr);
! static void CopyReadAttributesCSV(const char *delim, const char *null_print, char *quote,
! char *escape, int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr);
static Datum CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
Oid typioparam, int32 typmod, bool *isnull);
+ static char *CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
+ bool *isnull);
static void CopyAttributeOut(char *string, char *delim);
static void CopyAttributeOutCSV(char *string, char *delim, char *quote,
char *escape, bool force_quote);
static List *CopyGetAttnums(Relation rel, List *attnamelist);
static void limit_printout_length(StringInfo buf);
+ static bool DetectLineEnd(size_t bytesread, char *quote, char *escape);
+
/* Internal communications functions */
static void SendCopyBegin(bool binary, int natts);
***************
*** 159,175 ****
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static void CopyGetData(void *databuf, int datasize);
! static int CopyGetChar(void);
#define CopyGetEof() (fe_eof)
- static int CopyPeekChar(void);
- static void CopyDonePeek(int c, bool pickup);
static void CopySendInt32(int32 val);
static int32 CopyGetInt32(void);
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
/*
* Send copy start/stop messages for frontend copies. These have changed
--- 180,197 ----
static void CopySendString(const char *str);
static void CopySendChar(char c);
static void CopySendEndOfRow(bool binary);
! static int CopyGetData(void *databuf, int datasize);
#define CopyGetEof() (fe_eof)
static void CopySendInt32(int32 val);
static int32 CopyGetInt32(void);
static void CopySendInt16(int16 val);
static int16 CopyGetInt16(void);
+ /* byte scaning utils */
+ static char *scanTextLine(const char *s, char c, size_t len);
+ static char *scanCSVLine(const char *s, char c1, char c2, char c3, size_t len);
+ static char *scanTextAttr(const char *s, char c1, char c2, size_t len);
/*
* Send copy start/stop messages for frontend copies. These have changed
***************
*** 382,395 ****
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
*/
! static void
CopyGetData(void *databuf, int datasize)
{
switch (copy_dest)
{
case COPY_FILE:
! fread(databuf, datasize, 1, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
--- 404,422 ----
* It seems unwise to allow the COPY IN to complete normally in that case.
*
* NB: no data conversion is applied by these functions
+ *
+ * Returns: the number of bytes that were successfully read
+ * into the data buffer.
*/
! static int
CopyGetData(void *databuf, int datasize)
{
+ size_t bytesread = 0;
+
switch (copy_dest)
{
case COPY_FILE:
! bytesread = fread(databuf, 1, datasize, copy_file);
if (feof(copy_file))
fe_eof = true;
break;
***************
*** 401,406 ****
--- 428,435 ----
(errcode(ERRCODE_CONNECTION_FAILURE),
errmsg("unexpected EOF on client connection")));
}
+ bytesread += datasize; /* update the count of bytes that were
+ * read so far */
break;
case COPY_NEW_FE:
while (datasize > 0 && !fe_eof)
***************
*** 429,435 ****
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
--- 458,464 ----
case 'c': /* CopyDone */
/* COPY IN correctly terminated by frontend */
fe_eof = true;
! return bytesread;
case 'f': /* CopyFail */
ereport(ERROR,
(errcode(ERRCODE_QUERY_CANCELED),
***************
*** 459,598 ****
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
datasize -= avail;
}
break;
}
- }
-
- static int
- CopyGetChar(void)
- {
- int ch;
-
- switch (copy_dest)
- {
- case COPY_FILE:
- ch = getc(copy_file);
- break;
- case COPY_OLD_FE:
- ch = pq_getbyte();
- if (ch == EOF)
- {
- /* Only a \. terminator is legal EOF in old protocol */
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("unexpected EOF on client connection")));
- }
- break;
- case COPY_NEW_FE:
- {
- unsigned char cc;
-
- CopyGetData(&cc, 1);
- if (fe_eof)
- ch = EOF;
- else
- ch = cc;
- break;
- }
- default:
- ch = EOF;
- break;
- }
- if (ch == EOF)
- fe_eof = true;
- return ch;
- }
-
- /*
- * CopyPeekChar reads a byte in "peekable" mode.
- *
- * after each call to CopyPeekChar, a call to CopyDonePeek _must_
- * follow, unless EOF was returned.
- *
- * CopyDonePeek will either take the peeked char off the stream
- * (if pickup is true) or leave it on the stream (if pickup is false).
- */
- static int
- CopyPeekChar(void)
- {
- int ch;
-
- switch (copy_dest)
- {
- case COPY_FILE:
- ch = getc(copy_file);
- break;
- case COPY_OLD_FE:
- ch = pq_peekbyte();
- if (ch == EOF)
- {
- /* Only a \. terminator is legal EOF in old protocol */
- ereport(ERROR,
- (errcode(ERRCODE_CONNECTION_FAILURE),
- errmsg("unexpected EOF on client connection")));
- }
- break;
- case COPY_NEW_FE:
- {
- unsigned char cc;
-
- CopyGetData(&cc, 1);
- if (fe_eof)
- ch = EOF;
- else
- ch = cc;
- break;
- }
- default:
- ch = EOF;
- break;
- }
- if (ch == EOF)
- fe_eof = true;
- return ch;
- }
-
- static void
- CopyDonePeek(int c, bool pickup)
- {
- if (fe_eof)
- return; /* can't unget an EOF */
- switch (copy_dest)
- {
- case COPY_FILE:
- if (!pickup)
- {
- /* We don't want to pick it up - so put it back in there */
- ungetc(c, copy_file);
- }
- /* If we wanted to pick it up, it's already done */
- break;
- case COPY_OLD_FE:
- if (pickup)
- {
- /* We want to pick it up */
- (void) pq_getbyte();
- }
! /*
! * If we didn't want to pick it up, just leave it where it
! * sits
! */
! break;
! case COPY_NEW_FE:
! if (!pickup)
! {
! /* We don't want to pick it up - so put it back in there */
! copy_msgbuf->cursor--;
! }
! /* If we wanted to pick it up, it's already done */
! break;
! }
}
-
/*
* These functions do apply some data conversion
*/
--- 488,503 ----
avail = datasize;
pq_copymsgbytes(copy_msgbuf, databuf, avail);
databuf = (void *) ((char *) databuf + avail);
+ bytesread += avail; /* update the count of bytes that were
+ * read so far */
datasize -= avail;
}
break;
}
! return bytesread;
}
/*
* These functions do apply some data conversion
*/
***************
*** 968,980 ****
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attribute_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
--- 873,905 ----
}
/* Set up variables to avoid per-attribute overhead. */
! initStringInfo(&attr_buf);
initStringInfo(&line_buf);
line_buf_converted = false;
client_encoding = pg_get_client_encoding();
server_encoding = GetDatabaseEncoding();
+ /*
+ * check if the client encoding is one of the 5 encodings
+ * that are not supported as a server encodings.
+ */
+ switch (client_encoding)
+ {
+ case PG_SJIS:
+ case PG_BIG5:
+ case PG_GBK:
+ case PG_UHC:
+ case PG_GB18030:
+ client_encoding_only = true;
+ break;
+ default:
+ client_encoding_only = false;
+ }
+
+ if(!csv_mode)
+ escape = "\\"; /* default for text format */
+
copy_dest = COPY_FILE; /* default */
copy_file = NULL;
copy_msgbuf = NULL;
***************
*** 1105,1113 ****
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attribute_buf.data);
pfree(line_buf.data);
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
--- 1030,1039 ----
errmsg("could not write to file \"%s\": %m",
filename)));
}
! pfree(attr_buf.data);
pfree(line_buf.data);
+
/*
* Close the relation. If reading, we can release the AccessShareLock
* we got; if writing, we should hold the lock until end of
***************
*** 1345,1351 ****
{
bytea *outputbytes;
! outputbytes = DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
--- 1271,1278 ----
{
bytea *outputbytes;
! outputbytes =
! DatumGetByteaP(FunctionCall1(&out_functions[attnum - 1],
value));
/* We assume the result will not have been toasted */
CopySendInt32(VARSIZE(outputbytes) - VARHDRSZ);
***************
*** 1395,1404 ****
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attribute_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attribute_buf.data);
}
else
{
--- 1322,1331 ----
if (copy_attname)
{
/* error is relevant to a particular column */
! limit_printout_length(&attr_buf);
errcontext("COPY %s, line %d, column %s: \"%s\"",
copy_relname, copy_lineno, copy_attname,
! attr_buf.data);
}
else
{
***************
*** 1406,1411 ****
--- 1333,1340 ----
if (line_buf_converted ||
client_encoding == server_encoding)
{
+ /* Strip off the newline */
+ *(line_buf.data + line_buf.len - 1) = '\0';
limit_printout_length(&line_buf);
errcontext("COPY %s, line %d: \"%s\"",
copy_relname, copy_lineno,
***************
*** 1485,1491 ****
Oid in_func_oid;
Datum *values;
char *nulls;
- bool done = false;
bool isnull;
ResultRelInfo *resultRelInfo;
EState *estate = CreateExecutorState(); /* for ExecConstraints() */
--- 1414,1419 ----
***************
*** 1496,1501 ****
--- 1424,1434 ----
ExprContext *econtext; /* used for ExecEvalExpr for default atts */
MemoryContext oldcontext = CurrentMemoryContext;
ErrorContextCallback errcontext;
+ int *attr_offsets;
+ int null_print_len; /* length of null print */
+ bool no_more_data;
+ ListCell *cur;
+
tupDesc = RelationGetDescr(rel);
attr = tupDesc->attrs;
***************
*** 1607,1613 ****
}
}
! /* Prepare to catch AFTER triggers. */
AfterTriggerBeginQuery();
/*
--- 1540,1548 ----
}
}
! /*
! * Prepare to catch AFTER triggers.
! */
AfterTriggerBeginQuery();
/*
***************
*** 1671,1676 ****
--- 1606,1612 ----
values = (Datum *) palloc(num_phys_attrs * sizeof(Datum));
nulls = (char *) palloc(num_phys_attrs * sizeof(char));
+ attr_offsets = (int *) palloc(num_phys_attrs * sizeof(int));
/* Make room for a PARAM_EXEC value for domain constraint checks */
if (hasConstraints)
***************
*** 1691,1712 ****
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /* on input just throw the header line away */
! if (header_line)
{
! copy_lineno++;
! done = CopyReadLine(quote, escape) ;
}
! while (!done)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;
CHECK_FOR_INTERRUPTS();
- copy_lineno++;
-
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
--- 1627,1681 ----
errcontext.previous = error_context_stack;
error_context_stack = &errcontext;
! /*
! * initialize buffered scan variables.
! */
! if(csv_mode)
! {
! in_quote = false;
! last_was_esc = false;
! }
!
! null_print_len = strlen(null_print);
!
! /* Set up data buffer to hold a chunk of data */
! MemSet(input_buf, ' ', COPY_BUF_SIZE * sizeof(char));
! input_buf[COPY_BUF_SIZE] = '\0';
!
! no_more_data = false; /* no more input data to read from file or FE */
! line_done = true;
! buf_done = false;
!
! do
{
! size_t bytesread = 0;
!
! /* read a chunk of data into the buffer */
! if (!binary)
! {
! bytesread = CopyGetData(input_buf, COPY_BUF_SIZE);
! buf_done = false;
!
! /* set buffer pointers to beginning of the buffer */
! begloc = input_buf;
! buffer_index = 0;
}
! /*
! * continue if some bytes were read or if we didn't reach EOF. if we
! * both reached EOF _and_ no bytes were read, quit the loop we are
! * done
! */
! if (bytesread > 0 || !fe_eof || binary)
! {
!
! while (!buf_done)
{
bool skip_tuple;
Oid loaded_oid = InvalidOid;
CHECK_FOR_INTERRUPTS();
/* Reset the per-tuple exprcontext */
ResetPerTupleExprContext(estate);
***************
*** 1716,1746 ****
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
if (!binary)
{
- CopyReadResult result = NORMAL_ATTR;
- char *string;
- ListCell *cur;
/* Actually read the line into memory here */
! done = csv_mode ?
! CopyReadLine(quote, escape) : CopyReadLine(NULL, NULL);
/*
! * EOF at start of line means we're done. If we see EOF after
! * some characters, we act as though it was newline followed
! * by EOF, ie, process the line and then exit loop on next
! * iteration.
*/
! if (done && line_buf.len == 0)
break;
if (file_has_oids)
{
/* can't be in CSV mode here */
! string = CopyReadAttribute(delim, null_print,
! &result, &isnull);
if (isnull)
ereport(ERROR,
--- 1685,1729 ----
/* Initialize all values for row to NULL */
MemSet(values, 0, num_phys_attrs * sizeof(Datum));
MemSet(nulls, 'n', num_phys_attrs * sizeof(char));
+ /* reset attribute pointers */
+ MemSet(attr_offsets, 0, num_phys_attrs * sizeof(int));
if (!binary)
{
/* Actually read the line into memory here */
! line_done = csv_mode ?
! CopyReadLineCSV(bytesread, quote, escape) :
! CopyReadLineText(bytesread, escape);
/*
! * if finished processing data line - increment line count.
! * Otherwise, if eof is not yet reached, we skip att parsing
! * and read more data. But if eof _was_ reached it means
! * that the original last data line is defective and
! * we want to catch that error later on.
*/
! if (line_done)
! copy_lineno++;
! else if (!fe_eof || end_marker )
break;
+ if (header_line)
+ {
+ line_buf.len = 0; /* we can reset line buffer now. */
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ header_line = false;
+ continue;
+ }
+
if (file_has_oids)
{
+ char *oid_string;
+
/* can't be in CSV mode here */
! oid_string = CopyReadOidAttr(delim, null_print, null_print_len,
! &isnull);
if (isnull)
ereport(ERROR,
***************
*** 1750,1756 ****
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
--- 1733,1739 ----
{
copy_attname = "oid";
loaded_oid = DatumGetObjectId(DirectFunctionCall1(oidin,
! CStringGetDatum(oid_string)));
if (loaded_oid == InvalidOid)
ereport(ERROR,
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
***************
*** 1759,1792 ****
}
}
! /* Loop to read the user attributes on the line. */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
! /*
! * If prior attr on this line was ended by newline,
! * complain.
! */
! if (result != NORMAL_ATTR)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m]->attname))));
! if (csv_mode)
! {
! string = CopyReadAttributeCSV(delim, null_print, quote,
! escape, &result, &isnull);
! if (result == UNTERMINATED_FIELD)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unterminated CSV quoted field")));
! }
else
! string = CopyReadAttribute(delim, null_print,
! &result, &isnull);
if (csv_mode && isnull && force_notnull[m])
{
--- 1742,1770 ----
}
}
! /* parse all the attribute in the data line */
! if(csv_mode)
! CopyReadAttributesCSV(delim, null_print, quote, escape, null_print_len,
! nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
! else
! CopyReadAttributesText(delim, escape, null_print, null_print_len,
! nulls, attnumlist, attr_offsets, num_phys_attrs, attr);
!
! /*
! * Loop to read the user attributes on the line.
! */
foreach(cur, attnumlist)
{
int attnum = lfirst_int(cur);
int m = attnum - 1;
+ char *string;
! string = attr_buf.data + attr_offsets[m];
! if (nulls[m] == ' ')
! isnull = false;
else
! isnull = true;
if (csv_mode && isnull && force_notnull[m])
{
***************
*** 1806,1823 ****
copy_attname = NULL;
}
}
-
- /*
- * Complain if there are more fields on the input line.
- *
- * Special case: if we're reading a zero-column table, we won't
- * yet have called CopyReadAttribute() at all; so no error if
- * line is empty.
- */
- if (result == NORMAL_ATTR && line_buf.len != 0)
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("extra data after last expected column")));
}
else
{
--- 1784,1789 ----
***************
*** 1828,1834 ****
fld_count = CopyGetInt16();
if (CopyGetEof() || fld_count == -1)
{
! done = true;
break;
}
--- 1794,1801 ----
fld_count = CopyGetInt16();
if (CopyGetEof() || fld_count == -1)
{
! buf_done = true;
! no_more_data = true;
break;
}
***************
*** 1885,1891 ****
nulls[defmap[i]] = ' ';
}
! /* Next apply any domain constraints */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
--- 1852,1860 ----
nulls[defmap[i]] = ' ';
}
! /*
! * Next apply any domain constraints
! */
if (hasConstraints)
{
ParamExecData *prmdata = &econtext->ecxt_param_exec_vals[0];
***************
*** 1912,1924 ****
}
}
! /* And now we can form the input tuple. */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /* Triggers and stuff need to be invoked in query context. */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
--- 1881,1897 ----
}
}
! /*
! * And now we can form the input tuple.
! */
tuple = heap_formtuple(tupDesc, values, nulls);
if (oids && file_has_oids)
HeapTupleSetOid(tuple, loaded_oid);
! /*
! * Triggers and stuff need to be invoked in query context.
! */
MemoryContextSwitchTo(oldcontext);
skip_tuple = false;
***************
*** 1945,1955 ****
/* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /* Check the constraints of the tuple */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /* OK, store the tuple and create index entries for it */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
--- 1918,1932 ----
/* Place tuple in tuple slot */
ExecStoreTuple(tuple, slot, InvalidBuffer, false);
! /*
! * Check the constraints of the tuple
! */
if (rel->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
! /*
! * OK, store the tuple and create index entries for it
! */
simple_heap_insert(rel, tuple);
if (resultRelInfo->ri_NumIndices > 0)
***************
*** 1958,1976 ****
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
}
! /* Done, clean up */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /* Execute AFTER STATEMENT insertion triggers */
ExecASInsertTriggers(estate, resultRelInfo);
! /* Handle queued AFTER triggers */
AfterTriggerEndQuery(estate);
pfree(values);
pfree(nulls);
--- 1935,1972 ----
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, tuple);
}
+
+ line_buf.len = 0; /* we can reset line buffer now. */
+ line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
+ } /* end while(!buf_done) */
+ } /* end if (bytesread > 0 || !fe_eof) */
+ else
+ /* no bytes read, end of data */
+ {
+ no_more_data = true;
}
+ } while (!no_more_data);
! /*
! * Done, clean up
! */
error_context_stack = errcontext.previous;
MemoryContextSwitchTo(oldcontext);
! /*
! * Execute AFTER STATEMENT insertion triggers
! */
ExecASInsertTriggers(estate, resultRelInfo);
! /*
! * Handle queued AFTER triggers
! */
AfterTriggerEndQuery(estate);
+ pfree(attr_offsets);
+
pfree(values);
pfree(nulls);
***************
*** 1990,2262 ****
/*
! * Read the next input line and stash it in line_buf, with conversion to
! * server encoding.
! *
! * Result is true if read was terminated by EOF, false if terminated
! * by newline.
*/
static bool
! CopyReadLine(char * quote, char * escape)
{
! bool result;
! bool change_encoding = (client_encoding != server_encoding);
! int c;
! int mblen;
! int j;
! unsigned char s[2];
char *cvt;
! bool in_quote = false, last_was_esc = false, csv_mode = false;
! char quotec = '\0', escapec = '\0';
!
! if (quote)
! {
! csv_mode = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
!
!
! s[1] = 0;
!
! /* reset line_buf to empty */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
/* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /* set default status */
! result = false;
/*
! * In this loop we only care for detecting newlines (\r and/or \n) and
! * the end-of-copy marker (\.).
! *
! * In Text mode, for backwards compatibility we allow
! * backslashes to escape newline characters. Backslashes other than
! * the end marker get put into the line_buf, since CopyReadAttribute
! * does its own escape processing.
! *
! * In CSV mode, CR and NL inside q quoted field are just part of the
! * data value and are put in line_buf. We keep just enough state
! * to know if we are currently in a quoted field or not.
! *
! * These four characters, and only these four, are assumed the same in
! * frontend and backend encodings.
! *
! * We do not assume that second and later bytes of a frontend
! * multibyte character couldn't look like ASCII characters.
*/
! for (;;)
{
! c = CopyGetChar();
! if (c == EOF)
{
! result = true;
! break;
}
! if (csv_mode)
{
/*
! * Dealing with quotes and escapes here is mildly tricky. If the
! * quote char is also the escape char, there's no problem - we
! * just use the char as a toggle. If they are different, we need
! * to ensure that we only take account of an escape inside a quoted
! * field and immediately preceding a quote char, and not the
! * second in a escape-escape sequence.
*/
! if (in_quote && c == escapec)
! last_was_esc = ! last_was_esc;
! if (c == quotec && ! last_was_esc)
! in_quote = ! in_quote;
! if (c != escapec)
! last_was_esc = false;
/*
! * updating the line count for embedded CR and/or LF chars is
! * necessarily a little fragile - this test is probably about
! * the best we can do.
*/
! if (in_quote && c == (eol_type == EOL_CR ? '\r' : '\n'))
! copy_lineno++;
! }
! if (!in_quote && c == '\r')
{
! if (eol_type == EOL_NL)
{
! if (! csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in CSV data"),
! errhint("Use quoted CSV field to represent carriage return.")));
}
! /* Check for \r\n on first line, _and_ handle \r\n. */
! if (eol_type == EOL_UNKNOWN || eol_type == EOL_CRNL)
{
! int c2 = CopyPeekChar();
! if (c2 == '\n')
! {
! CopyDonePeek(c2, true); /* eat newline */
! eol_type = EOL_CRNL;
}
else
{
! /* found \r, but no \n */
if (eol_type == EOL_CRNL)
{
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal carriage return found in data"),
! errhint("Use \"\\r\" to represent carriage return.")));
else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted carriage return found in data"),
! errhint("Use quoted CSV field to represent carriage return.")));
!
}
/*
! * if we got here, it is the first line and we didn't
! * get \n, so put it back
*/
! CopyDonePeek(c2, false);
! eol_type = EOL_CR;
}
}
- break;
}
! if (!in_quote && c == '\n')
{
! if (eol_type == EOL_CR || eol_type == EOL_CRNL)
! {
! if (!csv_mode)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("literal newline found in data"),
! errhint("Use \"\\n\" to represent newline.")));
! else
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unquoted newline found in data"),
! errhint("Use quoted CSV field to represent newline.")));
!
}
- eol_type = EOL_NL;
- break;
- }
! if ((line_buf.len == 0 || !csv_mode) && c == '\\')
! {
! int c2;
!
! if (csv_mode)
! c2 = CopyPeekChar();
! else
! c2 = c = CopyGetChar();
! if (c2 == EOF)
{
! result = true;
! if (csv_mode)
! CopyDonePeek(c2, true);
break;
}
! if (c2 == '.')
! {
! if (csv_mode)
! CopyDonePeek(c2, true); /* allow keep calling GetChar() */
! if (eol_type == EOL_CRNL)
! {
! c = CopyGetChar();
! if (c == '\n')
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker does not match previous newline style")));
! if (c != '\r')
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("end-of-copy marker corrupt")));
}
- c = CopyGetChar();
- if (c != '\r' && c != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("end-of-copy marker corrupt")));
- if ((eol_type == EOL_NL && c != '\n') ||
- (eol_type == EOL_CRNL && c != '\n') ||
- (eol_type == EOL_CR && c != '\r'))
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("end-of-copy marker does not match previous newline style")));
/*
! * In protocol version 3, we should ignore anything after
! * \. up to the protocol end of copy data. (XXX maybe
! * better not to treat \. as special?)
*/
! if (copy_dest == COPY_NEW_FE)
{
! while (c != EOF)
! c = CopyGetChar();
}
! result = true; /* report EOF */
break;
}
! if (csv_mode)
! CopyDonePeek(c2, false); /* not a dot, so put it back */
else
! /* not EOF mark, so emit \ and following char literally */
! appendStringInfoCharMacro(&line_buf, '\\');
}
! appendStringInfoCharMacro(&line_buf, c);
/*
! * When client encoding != server, must be careful to read the
! * extra bytes of a multibyte character exactly, since the
! * encoding might not ensure they don't look like ASCII. When the
! * encodings are the same, we need not do this, since no server
! * encoding we use has ASCII-like following bytes.
*/
! if (change_encoding)
! {
! s[0] = c;
! mblen = pg_encoding_mblen(client_encoding, s);
! for (j = 1; j < mblen; j++)
! {
! c = CopyGetChar();
! if (c == EOF)
{
! result = true;
break;
}
- appendStringInfoCharMacro(&line_buf, c);
}
! if (result)
! break; /* out of outer loop */
}
- } /* end of outer loop */
! /* Done reading the line. Convert it to server encoding. */
! if (change_encoding)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
--- 1986,2424 ----
/*
! * Finds the next TEXT line that is in the input buffer and loads
! * it into line_buf. Returns an indication if the line that was read
! * is complete (if an unescaped line-end was encountered). If we
! * reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
*/
static bool
! CopyReadLineText(size_t bytesread, char *escape)
{
! int linesize;
! bool transcode = (client_encoding != server_encoding);
char *cvt;
! char escapec = '\0';
/* mark that encoding conversion hasn't occurred yet */
line_buf_converted = false;
! /*
! * set the escape char for text format ('\\' by default).
! */
! escapec = escape[0];
/*
! * Detect end of line type if not already detected.
*/
! if (eol_type == EOL_UNKNOWN)
{
! bool eol_detected = DetectLineEnd(bytesread, NULL, escape);
!
! if (!eol_detected)
{
! /* load entire input buffer into line buf, and quit */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! buf_done = true;
! return false;
! }
}
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /* if we started scanning from the 1st byte of the buffer */
! if (begloc == input_buf)
! {
! /* and had a CR in last byte of prev buf */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence
! * (linefeed)
! */
! if (*begloc == eol_ch[1])
{
/*
! * load that one linefeed byte and indicate we are done
! * with the data line
*/
+ appendBinaryStringInfo(&line_buf, begloc, 1);
+ buffer_index++;
+ begloc++;
+ esc_in_prevbuf = false;
+ cr_in_prevbuf = false;
+ return true;
+ }
+ }
! cr_in_prevbuf = false;
! }
! }
/*
! * (we need a loop so that if eol_ch is found, but prev ch is backslash,
! * we can search for the next eol_ch)
*/
! while (true)
! {
! /* reached end of buffer */
! if ((endloc = scanTextLine(begloc, eol_ch[0], bytesread - buffer_index)) == NULL)
! {
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! if (line_buf.len > 1)
{
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == escapec)
{
! esc_in_prevbuf = true;
!
! if (line_buf.len > 2)
! {
! last_ch--;
! if (*last_ch == escapec)
! esc_in_prevbuf = false;
}
! }
! else if (*last_ch == '\r')
{
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
! }
! }
! line_done = false;
! buf_done = true;
! break;
}
else
+ /* found the 1st eol ch in input_buf. */
{
! bool eol_found = true;
! bool eol_escaped = true;
!
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
if (eol_type == EOL_CRNL)
{
! /* check if there is a '\n' after the '\r' */
! if (*(endloc + 1) == '\n')
! {
! /* this is a line end */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
else
! /* just a CR, not a line end */
! eol_found = false;
}
/*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
*/
! if (buffer_index >= bytesread)
! buf_done = true;
!
! /*
! * Check if the 1st end of line ch is escaped.
! */
! if (endloc != input_buf) /* can we look 1 char back? */
! {
! if (*(endloc - 1) != escapec) /* prev char is not an escape */
! eol_escaped = false;
! else
! /* prev char is an escape */
! {
! if (endloc != (input_buf + 1)) /* can we look another
! * char back? */
! {
! /* it's a double escape char, so it's not an escape */
! if (*(endloc - 2) == escapec)
! eol_escaped = false;
! /* else it's a single escape char, so EOL is ascaped */
! }
! else
! {
! /* we need to check in the last buffer */
! if (esc_in_prevbuf)
! eol_escaped = false;
}
}
}
! else
! /*
! * this eol ch is first ch in buffer, check for escape in prev buf
! */
{
! if (!esc_in_prevbuf)
! eol_escaped = false;
}
! esc_in_prevbuf = false; /* reset variable */
! /*
! * if eol was found, and it isn't escaped, line is done
! */
! if ((eol_escaped == false) && eol_found)
{
! line_done = true;
break;
}
! else
! /* stay in the loop and process some more data. */
! line_done = false;
!
! } /* end of found eol_ch */
! }
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
! {
! cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
! line_buf.len);
! if (cvt != line_buf.data)
! {
! /* transfer converted data back to line_buf */
! line_buf.len = 0;
! line_buf.data[0] = '\0';
! line_buf.cursor = 0;
! appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
! }
! }
!
! /* indicate that conversion had occured */
! line_buf_converted = true;
!
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
!
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
! {
! fe_eof = true;
! /* we don't want to process a \. as data line, want to quit. */
! line_done = false;
! buf_done = true;
! }
!
! return line_done;
! }
!
! /*
! * Finds the next CSV line that is in the input buffer and loads
! * it into line_buf. Returns an indication if the line that was read
! * is complete (if an unescaped line-end was encountered). If we
! * reached the end of buffer before the whole line was written into the
! * line buffer then returns false.
! */
! static bool
! CopyReadLineCSV(size_t bytesread, char *quote, char *escape)
! {
! int linesize;
! char *cvt;
! char quotec = '\0',
! escapec = '\0';
! bool transcode = (client_encoding != server_encoding);
!
! /* mark that encoding conversion hasn't occurred yet */
! line_buf_converted = false;
!
! escapec = escape[0];
! quotec = quote[0];
!
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
!
! /*
! * Detect end of line type if not already detected.
! */
! if (eol_type == EOL_UNKNOWN)
! {
! bool eol_detected = DetectLineEnd(bytesread, quote, escape);
!
! if (!eol_detected)
! {
! /* load entire input buffer into line buf, and quit */
! appendBinaryStringInfo(&line_buf, input_buf, COPY_BUF_SIZE);
! line_done = false;
! buf_done = true;
!
! return line_done;
! }
! }
!
! /*
! * Special case: eol is CRNL, last byte of previous buffer was an
! * unescaped CR and 1st byte of current buffer is NL. We check for
! * that here.
! */
! if (eol_type == EOL_CRNL)
! {
! /* if we started scanning from the 1st byte of the buffer */
! if (begloc == input_buf)
! {
! /* and had a CR in last byte of prev buf */
! if (cr_in_prevbuf)
! {
! /*
! * if this 1st byte in buffer is 2nd byte of line end sequence
! * (linefeed)
! */
! if (*begloc == eol_ch[1])
! {
! /*
! * load that one linefeed byte and indicate we are done
! * with the data line
! */
! appendBinaryStringInfo(&line_buf, begloc, 1);
! buffer_index++;
! begloc++;
!
! line_done = true;
! esc_in_prevbuf = false;
! cr_in_prevbuf = false;
!
! return line_done;
! }
! }
!
! cr_in_prevbuf = false;
! }
}
/*
! * (we need a loop so that if eol_ch is found, but we are in quotes,
! * we can search for the next eol_ch)
*/
! while (true)
! {
! /* reached end of buffer */
! if ((endloc = scanCSVLine(begloc, eol_ch[0], escapec, quotec, bytesread - buffer_index)) == NULL)
! {
! linesize = COPY_BUF_SIZE - (begloc - input_buf);
! appendBinaryStringInfo(&line_buf, begloc, linesize);
!
! if (last_was_esc)
! esc_in_prevbuf = true;
!
! if (line_buf.len > 1)
! {
! char *last_ch = line_buf.data + line_buf.len - 1; /* before terminating \0 */
!
! if (*last_ch == '\r')
{
! if (eol_type == EOL_CRNL)
! cr_in_prevbuf = true;
}
! }
!
! line_done = false;
! buf_done = true;
break;
}
+ else
+ /* found 1st eol char in input_buf. */
+ {
+ bool eol_found = true;
! /*
! * Load that piece of data (potentially a data line) into the line buffer,
! * and update the pointers for the next scan.
! */
! linesize = endloc - begloc + 1;
! appendBinaryStringInfo(&line_buf, begloc, linesize);
! buffer_index += linesize;
! begloc = endloc + 1;
!
! /* end of line only if not in quotes */
! if(in_quote)
! {
! line_done = false;
! /* buf done, but still in quote */
! if (buffer_index >= bytesread)
! {
! buf_done = true;
! break;
! }
! }
! else
! {
! /* if dos eol, check for '\n' after the '\r' */
! if (eol_type == EOL_CRNL)
! {
! if (*(endloc + 1) == '\n')
! {
! /* this is a line end */
! appendBinaryStringInfo(&line_buf, begloc, 1); /* load that '\n' */
! buffer_index++;
! begloc++;
! }
else
! /* just a CR, not a line end */
! eol_found = false;
}
! /*
! * in some cases, this end of line char happens to be the
! * last character in the buffer. we need to catch that.
! */
! if (buffer_index >= bytesread)
! buf_done = true;
/*
! * if eol was found line is done
*/
! if (eol_found)
{
! line_done = true;
break;
}
}
! } /* end of found eol_ch */
}
! /*
! * Done reading the line. Convert it to server encoding.
! */
! if (transcode)
{
cvt = (char *) pg_client_to_server((unsigned char *) line_buf.data,
line_buf.len);
***************
*** 2265,2285 ****
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* Now it's safe to use the buffer in error messages */
line_buf_converted = true;
! return result;
}
/*
* Return decimal value for a hexadecimal digit
*/
static
! int GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
--- 2427,2480 ----
/* transfer converted data back to line_buf */
line_buf.len = 0;
line_buf.data[0] = '\0';
+ line_buf.cursor = 0;
appendBinaryStringInfo(&line_buf, cvt, strlen(cvt));
}
}
! /* indicate that conversion had occured */
line_buf_converted = true;
! /*
! * check if this line is an end marker -- "\."
! */
! end_marker = false;
!
! switch (eol_type)
! {
! case EOL_NL:
! if (!strcmp(line_buf.data, "\\.\n"))
! end_marker = true;
! break;
! case EOL_CR:
! if (!strcmp(line_buf.data, "\\.\r"))
! end_marker = true;
! break;
! case EOL_CRNL:
! if (!strcmp(line_buf.data, "\\.\r\n"))
! end_marker = true;
! break;
! case EOL_UNKNOWN:
! break;
! }
!
! if (end_marker)
! {
! fe_eof = true;
! /* we don't want to process a \. as data line, want to quit. */
! line_done = false;
! buf_done = true;
! }
!
! return line_done;
}
/*
* Return decimal value for a hexadecimal digit
*/
static
! int
! GetDecimalFromHex(char hex)
{
if (isdigit(hex))
return hex - '0';
***************
*** 2287,2344 ****
return tolower(hex) - 'a' + 10;
}
! /*----------
! * Read the value of a single attribute, performing de-escaping as needed.
! *
! * delim is the column delimiter string (must be just one byte for now).
! * null_print is the null marker string. Note that this is compared to
! * the pre-de-escaped input string.
! *
! * *result is set to indicate what terminated the read:
! * NORMAL_ATTR: column delimiter
! * END_OF_LINE: end of line
! * In either case, the string read up to the terminator is returned.
! *
! * *isnull is set true or false depending on whether the input matched
! * the null marker. Note that the caller cannot check this since the
! * returned string will be the post-de-escaping equivalent, which may
! * look the same as some valid data string.
! *----------
*/
! static char *
! CopyReadAttribute(const char *delim, const char *null_print,
! CopyReadResult *result, bool *isnull)
{
char c;
! char delimc = delim[0];
! int start_cursor = line_buf.cursor;
! int end_cursor;
! int input_len;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! /* set default status */
! *result = END_OF_LINE;
! for (;;)
{
! end_cursor = line_buf.cursor;
! if (line_buf.cursor >= line_buf.len)
! break;
! c = line_buf.data[line_buf.cursor++];
! if (c == delimc)
{
! *result = NORMAL_ATTR;
! break;
}
! if (c == '\\')
{
! if (line_buf.cursor >= line_buf.len)
! break;
! c = line_buf.data[line_buf.cursor++];
! switch (c)
{
case '0':
case '1':
--- 2482,2771 ----
return tolower(hex) - 'a' + 10;
}
! /*
! * Detected the eol type by looking at the first data row.
! * Possible eol types are NL, CR, or CRNL. If eol type was
! * detected, it is set and a boolean true is returned to
! * indicated detection was successful. If the first data row
! * is longer than the input buffer, we return false and will
! * try again in the next buffer.
*/
! static bool
! DetectLineEnd(size_t bytesread, char *quote, char *escape)
{
+ int index = 0;
char c;
! char quotec = '\0',
! escapec = '\0';
! bool csv = false;
! if (quote) /* CSV format */
! {
! csv = true;
! quotec = quote[0];
! escapec = escape[0];
! /* ignore special escape processing if it's the same as quotec */
! if (quotec == escapec)
! escapec = '\0';
! }
! while (index < COPY_BUF_SIZE)
! {
! c = input_buf[index];
! if(csv)
{
! if (in_quote && c == escapec)
! last_was_esc = !last_was_esc;
! if (c == quotec && !last_was_esc)
! in_quote = !in_quote;
! if (c != escapec)
! last_was_esc = false;
! }
!
! if (c == '\n')
{
! if(!csv || (csv && !in_quote))
! {
! eol_type = EOL_NL;
! eol_ch[0] = '\n';
! eol_ch[1] = '\0';
!
! in_quote = false;
! last_was_esc = false;
! return true;
! }
}
! if (c == '\r')
{
! if(!csv || (csv && !in_quote))
! {
! if (input_buf[index + 1] == '\n') /* always safe */
! {
! eol_type = EOL_CRNL;
! eol_ch[0] = '\r';
! eol_ch[1] = '\n';
! }
! else
! {
! eol_type = EOL_CR;
! eol_ch[0] = '\r';
! eol_ch[1] = '\0';
! }
!
! in_quote = false;
! last_was_esc = false;
! return true;
! }
! }
!
! index++;
! }
!
! return false;
! }
!
! /*
! * Read all TEXT attributes. Attributes are parsed from line_buf and
! * inserted (all at once) to attr_buf, while saving pointers to
! * each attribute's starting position.
! *
! * When this routine finishes execution both the nulls array and
! * the attr_offsets array are updated. The attr_offsets will include
! * the offset from the beginning of the attribute array of which
! * each attribute begins. If a specific attribute is not used for this
! * COPY command (ommitted from the column list), a value of 0 will be assigned.
! * For example: for table foo(a,b,c,d,e) and COPY foo(a,b,e)
! * attr_offsets may look something like this after this routine
! * returns: [0,20,0,0,55]. That means that column "a" value starts
! * at byte offset 0, "b" in 20 and "e" in 55, in attr_buf.
! *
! * In the attribute buffer (attr_buf) each attribute
! * is terminated with a '\0', and therefore by using the attr_offsets
! * array we could point to a beginning of an attribute and have it
! * behave as a C string, much like previously done in COPY.
! *
! * Another aspect to improving performance is reducing the frequency
! * of data load into buffers. The original COPY read attribute code
! * loaded a character at a time. In here we try to load a chunk of data
! * at a time. Usually a chunk will include a full data row
! * (unless we have an escaped delim). That effectively reduces the number of
! * loads by a factor of number of bytes per row. This improves performance
! * greatly, unfortunately it add more complexity to the code.
! *
! * Global participants in parsing logic:
! *
! * line_buf.cursor -- an offset from beginning of the line buffer
! * that indicates where we are about to begin the next scan. Note that
! * if we have WITH OIDS this cursor is already shifted past the first
! * OID attribute.
! *
! * attr_buf.cursor -- an offset from the beginning of the
! * attribute buffer that indicates where the current attribute begins.
! */
! static void
! CopyReadAttributesText(const char *delim, const char *escape, const char *null_print,
! int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr)
! {
! char delimc = delim[0]; /* delimiter character */
! char escapec = escape[0]; /* escape character */
! char *scan_start; /* pointer to line buffer for scan start. */
! char *scan_end; /* pointer to line buffer where char was found */
! int attr_pre_len; /* attr raw len, before processing escapes */
! int attr_post_len; /* current attr len after escaping */
! int m; /* attribute index being parsed */
! int bytes_remaining;/* num bytes remaining to be scanned in line buf */
! int chunk_start; /* offset to beginning of line chunk to load */
! int chunk_len; /* length of chunk of data to load to attr buf */
! int oct_val; /* byte value for octal escapes */
! int hex_val; /* byte value for hexadecimal escapes */
! char hexchar; /* char that appears after \x for hex values */
! int attnum; /* attribute number being parsed */
! ListCell *cur; /* cursor to attribute list used for this COPY */
! int attribute;
!
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
! /* cursor is now > 0 if we copy WITH OIDS */
! scan_start = line_buf.data + line_buf.cursor;
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! chunk_start = line_buf.cursor;
! chunk_len = 0;
! attr_pre_len = 0;
! attr_post_len = 0;
!
! /*
! * Scan through the line buffer to read all attributes data
! */
! while (line_buf.cursor < line_buf.len)
! {
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! if ((scan_end = scanTextAttr(scan_start, delimc, escapec, bytes_remaining))
! == NULL)
! {
! /* GOT TO END OF LINE BUFFER */
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* don't count eol char(s) in attr and chunk len calculation */
! if (eol_type == EOL_CRNL)
! {
! attr_pre_len += bytes_remaining - 2;
! chunk_len = line_buf.len - chunk_start - 2;
! }
! else
! {
! attr_pre_len += bytes_remaining - 1;
! chunk_len = line_buf.len - chunk_start - 1;
! }
!
! /* check if this is a NULL value or data value (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(line_buf.data + line_buf.len - attr_pre_len - 1, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! attr_offsets[m] = attr_buf.cursor;
!
!
! /* load the last chunk, the whole buffer in most cases */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! line_buf.cursor += attr_pre_len + 2; /* skip eol char and
! * '\0' to exit loop */
!
! if (lnext(cur) != NULL)
! {
! /*
! * For an empty data line, the previous COPY code will
! * fail it during the conversion stage. We can fail it here
! * already, but then we will fail the regression tests b/c
! * of a different error message. that's why we return so we
! * can get the same error message that regress expects. ahh...
! * this conditional is unnecessary and should be removed soon.
! */
! if (line_buf.len > 1)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("missing data for column \"%s\"",
! NameStr(attr[m + 1]->attname))));
! else
! return;
! }
! }
! else
! /* FOUND A DELIMITER OR ESCAPE */
! {
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! if (*scan_end == delimc) /* found a delimiter */
! {
! attnum = lfirst_int(cur);
! m = attnum - 1;
!
! /* (we don't include the delimiter ch in length) */
! attr_pre_len += scan_end - scan_start;
! /* (we don't include the delimiter ch in length) */
! attr_post_len += scan_end - scan_start;
!
! /* check if this is a null print or data (assumed NULL) */
! if (attr_pre_len == null_print_len
! &&
! strncmp(scan_end - attr_pre_len, null_print, attr_pre_len)
! == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* set the pointer to next attribute position */
! attr_offsets[m] = attr_buf.cursor;
!
! /*
! * update buffer cursors to our current location, +1 to skip
! * the delimc
! */
! line_buf.cursor = scan_end - line_buf.data + 1;
! attr_buf.cursor += attr_post_len + 1;
!
! /* prepare scan for next attr */
! scan_start = line_buf.data + line_buf.cursor;
! cur = lnext(cur);
! attr_pre_len = 0;
! attr_post_len = 0;
! }
! else
! /* found an escape character */
! {
! char nextc = *(scan_end + 1);
! char newc;
! int skip = 2;
!
! chunk_len = (scan_end - line_buf.data) - chunk_start + 1;
!
! /* load a chunk of data */
! appendBinaryStringInfo(&attr_buf, line_buf.data + chunk_start, chunk_len);
!
! switch (nextc)
{
case '0':
case '1':
***************
*** 2348,2445 ****
case '5':
case '6':
case '7':
! /* handle \013 */
! {
! int val;
! val = OCTVALUE(c);
! if (line_buf.cursor < line_buf.len)
! {
! c = line_buf.data[line_buf.cursor];
! if (ISOCTAL(c))
! {
! line_buf.cursor++;
! val = (val << 3) + OCTVALUE(c);
! if (line_buf.cursor < line_buf.len)
{
! c = line_buf.data[line_buf.cursor];
! if (ISOCTAL(c))
{
! line_buf.cursor++;
! val = (val << 3) + OCTVALUE(c);
}
}
! }
! }
! c = val & 0377;
! }
break;
case 'x':
/* Handle \x3F */
! if (line_buf.cursor < line_buf.len)
! {
! char hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! int val = GetDecimalFromHex(hexchar);
! line_buf.cursor++;
! if (line_buf.cursor < line_buf.len)
! {
! hexchar = line_buf.data[line_buf.cursor];
if (isxdigit(hexchar))
{
! line_buf.cursor++;
! val = (val << 4) + GetDecimalFromHex(hexchar);
! }
! }
! c = val & 0xff;
}
}
break;
case 'b':
! c = '\b';
break;
case 'f':
! c = '\f';
break;
case 'n':
! c = '\n';
break;
case 'r':
! c = '\r';
break;
case 't':
! c = '\t';
break;
case 'v':
! c = '\v';
break;
/*
! * in all other cases, take the char after '\'
! * literally
*/
}
! }
! appendStringInfoCharMacro(&attribute_buf, c);
}
! /* check whether raw input matched null marker */
! input_len = end_cursor - start_cursor;
! if (input_len == strlen(null_print) &&
! strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
*isnull = true;
! else
! *isnull = false;
! return attribute_buf.data;
! }
/*
! * Read the value of a single attribute in CSV mode,
* performing de-escaping as needed. Escaping does not follow the normal
* PostgreSQL text mode, but instead "standard" (i.e. common) CSV usage.
*
--- 2775,2948 ----
case '5':
case '6':
case '7':
! oct_val = OCTVALUE(nextc);
! nextc = *(scan_end + 2);
! /*
! * (no need for out bad access check since line if
! * buffered)
! */
! if (ISOCTAL(nextc))
{
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
! nextc = *(scan_end + 3);
! if (ISOCTAL(nextc))
{
! skip++;
! oct_val = (oct_val << 3) + OCTVALUE(nextc);
}
}
! newc = oct_val & 0377; /* the escaped byte value */
break;
case 'x':
/* Handle \x3F */
! hexchar = *(scan_end + 2);
if (isxdigit(hexchar))
{
! skip++;
! hex_val = GetDecimalFromHex(hexchar);
! hexchar = *(scan_end + 3);
if (isxdigit(hexchar))
{
! skip++;
! hex_val = (hex_val << 4) + GetDecimalFromHex(hexchar);
}
+ newc = hex_val & 0xff;
}
+ else /* "\x" with no hex value */
+ newc = nextc;
break;
case 'b':
! newc = '\b';
break;
case 'f':
! newc = '\f';
break;
case 'n':
! newc = '\n';
break;
case 'r':
! newc = '\r';
break;
case 't':
! newc = '\t';
break;
case 'v':
! newc = '\v';
! break;
! default:
! if (nextc == delimc)
! newc = delimc;
! else if (nextc == escapec)
! newc = escapec;
! else
! /* no escape sequence, take next char literaly */
! newc = nextc;
break;
+ }
+
+ /* update to current length, add escape and escaped chars */
+ attr_pre_len += scan_end - scan_start + 2;
+ /* update to current length, escaped char */
+ attr_post_len += scan_end - scan_start + 1;
/*
! * Need to get rid of the escape character. This is done by
! * loading the chunk up to including the escape character
! * into the attribute buffer. Then overwritting the backslash
! * with the escaped sequence or char, and continuing to scan
! * from *after* the char than is after the escape in line buf.
*/
+ *(attr_buf.data + attr_buf.len - 1) = newc;
+ line_buf.cursor = scan_end - line_buf.data + skip;
+ scan_start = scan_end + skip;
+ chunk_start = line_buf.cursor;
+ chunk_len = 0;
}
!
! } /* end delimiter/backslash */
!
! } /* end line buffer scan. */
!
! /*
! * Replace all delimiters with NULL for string termination.
! * NOTE: only delimiters (NOT necessarily all delimc) are replaced.
! * Example (delimc = '|'):
! * - Before: f 1 | f \| 2 | f 3
! * - After : f 1 \0 f | 2 \0 f 3
! */
! for (attribute = 1; attribute < num_phys_attrs; attribute++)
! {
! if (attr_offsets[attribute] != 0)
! *(attr_buf.data + attr_offsets[attribute] - 1) = '\0';
}
! }
!
!
! /*
! * Read a binary attribute
! */
! static Datum
! CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, int32 typmod,
! bool *isnull)
! {
! int32 fld_size;
! Datum result;
!
! fld_size = CopyGetInt32();
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
! if (fld_size == -1)
! {
*isnull = true;
! return (Datum) 0;
! }
! if (fld_size < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid field size")));
! /* reset attr_buf to empty, and load raw data in it */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! enlargeStringInfo(&attr_buf, fld_size);
!
! CopyGetData(attr_buf.data, fld_size);
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
+ attr_buf.len = fld_size;
+ attr_buf.data[fld_size] = '\0';
+
+ /* Call the column type's binary input converter */
+ result = FunctionCall3(flinfo,
+ PointerGetDatum(&attr_buf),
+ ObjectIdGetDatum(typioparam),
+ Int32GetDatum(typmod));
+
+ /* Trouble if it didn't eat the whole buffer */
+ if (attr_buf.cursor != attr_buf.len)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
+ errmsg("incorrect binary data format")));
+
+ *isnull = false;
+ return result;
+ }
/*
! * Read all the attributes of the data line in CSV mode,
* performing de-escaping as needed. Escaping does not follow the normal
* PostgreSQL text mode, but instead "standard" (i.e. common) CSV usage.
*
***************
*** 2448,2472 ****
*
* null_print is the null marker string. Note that this is compared to
* the pre-de-escaped input string (thus if it is quoted it is not a NULL).
- *
- * *result is set to indicate what terminated the read:
- * NORMAL_ATTR: column delimiter
- * END_OF_LINE: end of line
- * UNTERMINATED_FIELD no quote detected at end of a quoted field
- *
- * In any case, the string read up to the terminator (or end of file)
- * is returned.
- *
- * *isnull is set true or false depending on whether the input matched
- * the null marker. Note that the caller cannot check this since the
- * returned string will be the post-de-escaping equivalent, which may
- * look the same as some valid data string.
*----------
*/
!
! static char *
! CopyReadAttributeCSV(const char *delim, const char *null_print, char *quote,
! char *escape, CopyReadResult *result, bool *isnull)
{
char delimc = delim[0];
char quotec = quote[0];
--- 2951,2962 ----
*
* null_print is the null marker string. Note that this is compared to
* the pre-de-escaped input string (thus if it is quoted it is not a NULL).
*----------
*/
! static void
! CopyReadAttributesCSV(const char *delim, const char *null_print, char *quote,
! char *escape, int null_print_len, char *nulls, List *attnumlist,
! int *attr_offsets, int num_phys_attrs, Form_pg_attribute *attr)
{
char delimc = delim[0];
char quotec = quote[0];
***************
*** 2477,2502 ****
int input_len;
bool in_quote = false;
bool saw_quote = false;
! /* reset attribute_buf to empty */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
- /* set default status */
- *result = END_OF_LINE;
for (;;)
{
end_cursor = line_buf.cursor;
if (line_buf.cursor >= line_buf.len)
break;
c = line_buf.data[line_buf.cursor++];
/* unquoted field delimiter */
if (!in_quote && c == delimc)
{
! *result = NORMAL_ATTR;
! break;
}
/* start of quoted field (or part of field) */
--- 2967,3047 ----
int input_len;
bool in_quote = false;
bool saw_quote = false;
+ int attnum; /* attribute number being parsed */
+ int m; /* attribute index being parsed */
+ ListCell *cur; /* cursor to attribute list used for this COPY */
! /*
! * init variables for attribute scan
! */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! cur = list_head(attnumlist);
! attnum = lfirst_int(cur);
! m = attnum - 1;
! input_len = 0;
for (;;)
{
end_cursor = line_buf.cursor;
+
+ /* finished processing attributes in line */
if (line_buf.cursor >= line_buf.len)
+ {
+ /* check whether raw input matched null marker */
+ input_len = end_cursor - start_cursor;
+ if (!saw_quote && input_len == null_print_len &&
+ strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
+ nulls[m] = 'n';
+ else
+ nulls[m] = ' ';
+
+ if (in_quote)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("unterminated CSV quoted field")));
+
+ if (lnext(cur) != NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+ errmsg("missing data for column \"%s\"",
+ NameStr(attr[m + 1]->attname))));
break;
+ }
+
c = line_buf.data[line_buf.cursor++];
/* unquoted field delimiter */
if (!in_quote && c == delimc)
{
! /* check whether raw input matched null marker */
! input_len = end_cursor - start_cursor;
! if (!saw_quote && input_len == null_print_len &&
! strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
! nulls[m] = 'n';
! else
! nulls[m] = ' ';
!
! /* terminate attr string with '\0' */
! appendStringInfoCharMacro(&attr_buf, '\0');
! attr_buf.cursor++;
!
! /* setup next attribute scan */
! cur = lnext(cur);
!
! if (cur == NULL)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("extra data after last expected column")));
!
! attnum = lfirst_int(cur);
! m = attnum - 1;
! attr_offsets[m] = attr_buf.cursor;
! start_cursor = line_buf.cursor;
! continue;
}
/* start of quoted field (or part of field) */
***************
*** 2520,2527 ****
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attribute_buf, nextc);
line_buf.cursor++;
continue;
}
}
--- 3065,3073 ----
if (nextc == escapec || nextc == quotec)
{
! appendStringInfoCharMacro(&attr_buf, nextc);
line_buf.cursor++;
+ attr_buf.cursor++;
continue;
}
}
***************
*** 2537,2616 ****
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attribute_buf, c);
}
- if (in_quote)
- *result = UNTERMINATED_FIELD;
-
- /* check whether raw input matched null marker */
- input_len = end_cursor - start_cursor;
- if (!saw_quote && input_len == strlen(null_print) &&
- strncmp(&line_buf.data[start_cursor], null_print, input_len) == 0)
- *isnull = true;
- else
- *isnull = false;
-
- return attribute_buf.data;
}
/*
! * Read a binary attribute
*/
! static Datum
! CopyReadBinaryAttribute(int column_no, FmgrInfo *flinfo,
! Oid typioparam, int32 typmod,
bool *isnull)
{
! int32 fld_size;
! Datum result;
!
! fld_size = CopyGetInt32();
! if (CopyGetEof())
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("unexpected EOF in COPY data")));
! if (fld_size == -1)
! {
! *isnull = true;
! return (Datum) 0;
}
! if (fld_size < 0)
! ereport(ERROR,
! (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
! errmsg("invalid field size")));
! /* reset attribute_buf to empty, and load raw data in it */
! attribute_buf.len = 0;
! attribute_buf.data[0] = '\0';
! attribute_buf.cursor = 0;
! enlargeStringInfo(&attribute_buf, fld_size);
- CopyGetData(attribute_buf.data, fld_size);
- if (CopyGetEof())
- ereport(ERROR,
- (errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
- errmsg("unexpected EOF in COPY data")));
! attribute_buf.len = fld_size;
! attribute_buf.data[fld_size] = '\0';
! /* Call the column type's binary input converter */
! result = FunctionCall3(flinfo,
! PointerGetDatum(&attribute_buf),
! ObjectIdGetDatum(typioparam),
! Int32GetDatum(typmod));
- /* Trouble if it didn't eat the whole buffer */
- if (attribute_buf.cursor != attribute_buf.len)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
- errmsg("incorrect binary data format")));
- *isnull = false;
- return result;
- }
/*
* Send text representation of one attribute, with conversion and escaping
--- 3083,3149 ----
in_quote = false;
continue;
}
! appendStringInfoCharMacro(&attr_buf, c);
! attr_buf.cursor++;
}
}
/*
! * Read the first attribute. This is mainly used to maintain support
! * for an OID column. All the rest of the columns will be read at once with
! * CopyReadAttributesText.
*/
! static char *
! CopyReadOidAttr(const char *delim, const char *null_print, int null_print_len,
bool *isnull)
{
! char delimc = delim[0];
! char *start_loc = line_buf.data + line_buf.cursor;
! char *end_loc;
! int attr_len = 0;
! int bytes_remaining;
!
! /* reset attribute buf to empty */
! attr_buf.len = 0;
! attr_buf.data[0] = '\0';
! attr_buf.cursor = 0;
!
! /* # of bytes that were not yet processed in this line */
! bytes_remaining = line_buf.len - line_buf.cursor;
!
! /* got to end of line */
! if ((end_loc = scanTextLine(start_loc, delimc, bytes_remaining)) == NULL)
! {
! attr_len = bytes_remaining - 1; /* don't count '\n' in len calculation */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 2; /* skip '\n' and '\0' */
}
! else
! /* found a delimiter */
! {
! /*
! * (we don't care if delim was preceded with a backslash, because it's
! * an invalid OID anyway)
! */
! attr_len = end_loc - start_loc; /* we don't include the delimiter ch */
! appendBinaryStringInfo(&attr_buf, start_loc, attr_len);
! line_buf.cursor += attr_len + 1;
! }
! /* check whether raw input matched null marker */
! if (attr_len == null_print_len && strncmp(start_loc, null_print, attr_len) == 0)
! *isnull = true;
! else
! *isnull = false;
! return attr_buf.data;
! }
/*
* Send text representation of one attribute, with conversion and escaping
***************
*** 2800,2802 ****
--- 3333,3436 ----
return attnums;
}
+
+ /*
+ * The following are custom versions of the string function strchr().
+ * As opposed to the original strchr which searches through
+ * a string until the target character is found, or a NULL is
+ * found, this version will not return when a NULL is found.
+ * Instead it will search through a pre-defined length of
+ * bytes and will return only if the target character(s) is reached.
+ *
+ * If our client encoding is not a supported server encoding, we
+ * know that it is not safe to look at each character as trailing
+ * byte in a multibyte character may be a 7-bit ASCII equivalent.
+ * Therefore we use pg_encoding_mblen to skip to the end of the
+ * character.
+ *
+ * returns:
+ * pointer to c - if c is located within the string.
+ * NULL - if c was not found in specified length of search. Note:
+ * this DOESN'T mean that a '\0' was reached.
+ */
+ char *
+ scanTextLine(const char *s, char eol, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only && !line_buf_converted)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ for (start = s; *s != eol && s < start + len; s += mblen)
+ mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ return ((*s == eol) ? (char *) s : NULL);
+ }
+ else
+ return memchr(s,eol,len);
+ }
+
+
+ char *
+ scanCSVLine(const char *s, char eol, char escapec, char quotec, size_t len)
+ {
+ const char *start;
+
+ if (client_encoding_only && !line_buf_converted)
+ {
+ int mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+
+ for (start = s; *s != eol && s < start + len; s += mblen)
+ {
+ if (in_quote && *s == escapec)
+ last_was_esc = !last_was_esc;
+ if (*s == quotec && !last_was_esc)
+ in_quote = !in_quote;
+ if (*s != escapec)
+ last_was_esc = false;
+
+ mblen = pg_encoding_mblen(client_encoding, (unsigned char*)s);
+ }
+ }
+ else
+ /* safe to scroll byte by byte */
+ {
+ for (start = s; *s != eol && s < start + len; s++)
+ {
+ if (in_quote && *s == escapec)
+ last_was_esc = !last_was_esc;
+ if (*s == quotec && !last_was_esc)
+ in_quote = !in_quote;
+ if (*s != escapec)
+ last_was_esc = false;
+ }
+
+ }
+
+ if(*s != escapec) /* found eol_ch */
+ last_was_esc = false;
+
+ return ((*s == eol) ? (char *) s : NULL);
+ }
+
+ /*
+ * Scan for 1 of 2 characters necessary for attribute parsing. No need
+ * for checking for multi-byte characters since conversion already
+ * happened.
+ */
+ char *
+ scanTextAttr(const char *s, char c1, char c2, size_t len)
+ {
+ const char *start;
+
+ for (start = s; *s != c1 && *s != c2 && s < start + len; s++)
+ ;
+
+ return (*s != '\0' ? (char *) s : NULL);
+ }
+
+
+
+
+
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-08-06 21:04 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 02:54 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 03:25 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 3 replies; 52+ messages in thread
From: Tom Lane @ 2005-08-06 21:04 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
"Alon Goldshuv" <agoldshuv@greenplum.com> writes:
> New patch attached. It includes very minor changes. These are changes that
> were committed to CVS 3 weeks ago (copy.c 1.247) which I missed in the
> previous patch.
I've applied this with (rather extensive) revisions. I didn't like what
you had done with the control structure --- loading the input buffer
only at the outermost loop level was a bad design choice IMHO. You had
sprinkled the code with an unreasonable number of special cases in order
to try to cope with the effects of that mistake, but there were lots
of problems still left. Some of the bugs I noticed:
* Broke old-protocol COPY, since that has no provision for stopping at
the EOF marker except by parsing the data carefully to start with. The
backend would just hang up unless the total data size chanced to be a
multiple of 64K.
* Subtle change in interpretation of \. EOF marker (the existing code
will recognize it even when not at start of line).
* Seems to have thrown away detection of newline format discrepancies.
* Fails for zero-column tables.
* Broke display of column values during error context callback (would
always show the first column contents no matter which one is being
complained of).
* DetectLineEnd mistakenly assumes CR mode if very last character of first
bufferload is CR; need to reserve judgment until next char is available.
* DetectLineEnd fails to account for backslashed control characters,
so it will e.g. accept \ followed by \n as determining the newline
style.
* Fails to apply encoding conversion if first line exceeds copy buf
size, because when DetectLineEnd fails the quick-exit path doesn't do
it.
* There seem to be several bugs associated with the fact that input_buf[]
always has 64K of data in it even when EOF has been reached on the
input. One example:
echo -n 123 >zzz1
psql> create temp table t1(f1 text);
psql> copy t1 from '/home/tgl/zzz1';
psql> select * from t1;
hmm ... where'd that 64K of whitespace come from?
I rewrote the patch in a way that retained most of the speedups without
changing the basic control structure (except for replacing multiple
CopyReadAttribute calls with one CopyReadAttributes call per line).
I had some difficulty in generating test cases that weren't largely
I/O-bound, but AFAICT the patch as applied is about the same speed
as what you submitted.
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-07 02:33 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-08-07 02:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Tom,
Thanks for finding the bugs and reworking things.
> I had some difficulty in generating test cases that weren't largely
> I/O-bound, but AFAICT the patch as applied is about the same speed
> as what you submitted.
You achieve the important objective of knocking the parsing stage down a
lot, but your parsing code is actually about 20% slower than Alon's.
Before your patch:
Time: 14205.606 ms
With your patch:
Time: 10565.374 ms
With Alon's patch:
Time: 10289.845 ms
The parsing part of the code in your version is slower, but as a percentage
of the total it's hidden. The loss of 0.3 seconds on 143MB means:
- If parsing takes a total of 0.9 seconds, the parsing rate is 160MB/s
(143/0.9)
- If we add another 0.3 seconds to parsing to bring it to 1.2, then the
parsing rate becomes 120MB/s
When we improve the next stages of the processing (attribute conversion,
write-to disk), this will stand out a lot more. Our objective is to get the
COPY rate *much* faster than the current poky rate of 14MB/s (after this
patch).
- Luke
On 8/6/05 2:04 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
> "Alon Goldshuv" <agoldshuv@greenplum.com> writes:
>> New patch attached. It includes very minor changes. These are changes that
>> were committed to CVS 3 weeks ago (copy.c 1.247) which I missed in the
>> previous patch.
>
> I've applied this with (rather extensive) revisions. I didn't like what
> you had done with the control structure --- loading the input buffer
> only at the outermost loop level was a bad design choice IMHO. You had
> sprinkled the code with an unreasonable number of special cases in order
> to try to cope with the effects of that mistake, but there were lots
> of problems still left. Some of the bugs I noticed:
>
> * Broke old-protocol COPY, since that has no provision for stopping at
> the EOF marker except by parsing the data carefully to start with. The
> backend would just hang up unless the total data size chanced to be a
> multiple of 64K.
>
> * Subtle change in interpretation of \. EOF marker (the existing code
> will recognize it even when not at start of line).
>
> * Seems to have thrown away detection of newline format discrepancies.
>
> * Fails for zero-column tables.
>
> * Broke display of column values during error context callback (would
> always show the first column contents no matter which one is being
> complained of).
>
> * DetectLineEnd mistakenly assumes CR mode if very last character of first
> bufferload is CR; need to reserve judgment until next char is available.
>
> * DetectLineEnd fails to account for backslashed control characters,
> so it will e.g. accept \ followed by \n as determining the newline
> style.
>
> * Fails to apply encoding conversion if first line exceeds copy buf
> size, because when DetectLineEnd fails the quick-exit path doesn't do
> it.
>
> * There seem to be several bugs associated with the fact that input_buf[]
> always has 64K of data in it even when EOF has been reached on the
> input. One example:
> echo -n 123 >zzz1
> psql> create temp table t1(f1 text);
> psql> copy t1 from '/home/tgl/zzz1';
> psql> select * from t1;
> hmm ... where'd that 64K of whitespace come from?
>
> I rewrote the patch in a way that retained most of the speedups without
> changing the basic control structure (except for replacing multiple
> CopyReadAttribute calls with one CopyReadAttributes call per line).
>
> I had some difficulty in generating test cases that weren't largely
> I/O-bound, but AFAICT the patch as applied is about the same speed
> as what you submitted.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-08-07 04:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 05:21 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-10 04:48 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 2 replies; 52+ messages in thread
From: Tom Lane @ 2005-08-07 04:08 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
"Luke Lonergan" <llonergan@greenplum.com> writes:
>> I had some difficulty in generating test cases that weren't largely
>> I/O-bound, but AFAICT the patch as applied is about the same speed
>> as what you submitted.
> You achieve the important objective of knocking the parsing stage down a
> lot, but your parsing code is actually about 20% slower than Alon's.
I would like to see the exact test case you are using to make this
claim; the tests I did suggested my code is the same speed or faster.
The particular test case I was using was the "tenk1" data from the
regression database, duplicated out to about 600K rows so as to run
long enough to measure with some degree of repeatability.
As best I can tell, my version of CopyReadAttributes is significantly
quicker than Alon's, approximately balancing out the fact that my
version of CopyReadLine is slower. I did the latter first, and would
now be tempted to rewrite it in the same style as CopyReadAttributes,
ie one pass of memory-to-memory copy using pointers rather than buffer
indexes.
BTW, late today I figured out a way to get fairly reproducible
non-I/O-bound numbers about COPY FROM: use a trigger that suppresses
the actual inserts, thus:
create table foo ...
create function noway() returns trigger as
'begin return null; end' language plpgsql;
create trigger noway before insert on foo
for each row execute procedure noway();
then repeat:
copy foo from '/tmp/foo.data';
If the source file is not too large to fit in kernel disk cache, then
after the first iteration there is no I/O at all. I got numbers
that were reproducible within less than 1%, as opposed to 5% or more
variation when the thing was partially I/O bound. Pretty useless in the
real world, of course, but great for timing COPY's data-pushing.
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-07 05:21 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-10 00:41 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
1 sibling, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-08-07 05:21 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Tom,
On 8/6/05 9:08 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
> "Luke Lonergan" <llonergan@greenplum.com> writes:
>>> I had some difficulty in generating test cases that weren't largely
>>> I/O-bound, but AFAICT the patch as applied is about the same speed
>>> as what you submitted.
>
>> You achieve the important objective of knocking the parsing stage down a
>> lot, but your parsing code is actually about 20% slower than Alon's.
>
> I would like to see the exact test case you are using to make this
> claim; the tests I did suggested my code is the same speed or faster.
I showed mine - you show yours :-) Apparently our e-mail crossed.
> As best I can tell, my version of CopyReadAttributes is significantly
> quicker than Alon's, approximately balancing out the fact that my
> version of CopyReadLine is slower. I did the latter first, and would
> now be tempted to rewrite it in the same style as CopyReadAttributes,
> ie one pass of memory-to-memory copy using pointers rather than buffer
> indexes.
See previous timings - looks like Alon's parsing is substantially faster.
However, I'd like him to confirm by running with the "shunt" placed at
different stages, in this case between parse and attribute conversion (not
attribute parse).
> BTW, late today I figured out a way to get fairly reproducible
> non-I/O-bound numbers about COPY FROM: use a trigger that suppresses
> the actual inserts, thus:
>
> create table foo ...
> create function noway() returns trigger as
> 'begin return null; end' language plpgsql;
> create trigger noway before insert on foo
> for each row execute procedure noway();
> then repeat:
> copy foo from '/tmp/foo.data';
Cool! That's a better way than hacking code and inserting shunts.
Alon will likely hit this tomorrow.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 05:21 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-08-10 00:41 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-10 01:01 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 52+ messages in thread
From: Alon Goldshuv @ 2005-08-10 00:41 UTC (permalink / raw)
To: pgsql-patches@postgresql.org
I did some performance checks after the recent code commit.
The good news is that the parsing speed of COPY is now MUCH faster, which is
great. It is about 5 times faster - about 100MB/sec on my machine
(previously 20MB/sec at best, usually less).
The better news is that my original patch parsing speed reaches 120MB/sec,
about 20MB/sec faster than the code that's now in CVS. This can be
significant for the long scheme of things and for large data sets. Maybe we
can improve the current code a bit more to reach this number.
I performed those measurement by executing *only the parsing logic* of the
COPY pipeline. All data conversion (functioncall3(string...)) and tuple
handling (form_heaptuple etc...) and insertion were manually disabled. So
the only code measured is reading from disk and parsing to the attribute
level.
Cheers,
Alon.
On 8/7/05 1:21 AM, "Luke Lonergan" <llonergan@greenplum.com> wrote:
> Tom,
>
> On 8/6/05 9:08 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
>
>> "Luke Lonergan" <llonergan@greenplum.com> writes:
>>>> I had some difficulty in generating test cases that weren't largely
>>>> I/O-bound, but AFAICT the patch as applied is about the same speed
>>>> as what you submitted.
>>
>>> You achieve the important objective of knocking the parsing stage down a
>>> lot, but your parsing code is actually about 20% slower than Alon's.
>>
>> I would like to see the exact test case you are using to make this
>> claim; the tests I did suggested my code is the same speed or faster.
>
> I showed mine - you show yours :-) Apparently our e-mail crossed.
>
>> As best I can tell, my version of CopyReadAttributes is significantly
>> quicker than Alon's, approximately balancing out the fact that my
>> version of CopyReadLine is slower. I did the latter first, and would
>> now be tempted to rewrite it in the same style as CopyReadAttributes,
>> ie one pass of memory-to-memory copy using pointers rather than buffer
>> indexes.
>
> See previous timings - looks like Alon's parsing is substantially faster.
> However, I'd like him to confirm by running with the "shunt" placed at
> different stages, in this case between parse and attribute conversion (not
> attribute parse).
>
>> BTW, late today I figured out a way to get fairly reproducible
>> non-I/O-bound numbers about COPY FROM: use a trigger that suppresses
>> the actual inserts, thus:
>>
>> create table foo ...
>> create function noway() returns trigger as
>> 'begin return null; end' language plpgsql;
>> create trigger noway before insert on foo
>> for each row execute procedure noway();
>> then repeat:
>> copy foo from '/tmp/foo.data';
>
> Cool! That's a better way than hacking code and inserting shunts.
>
> Alon will likely hit this tomorrow.
>
> - Luke
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 05:21 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-10 00:41 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-08-10 01:01 ` Andrew Dunstan <andrew@dunslane.net>
2005-08-10 04:39 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Andrew Dunstan @ 2005-08-10 01:01 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Alon Goldshuv wrote:
>I performed those measurement by executing *only the parsing logic* of the
>COPY pipeline. All data conversion (functioncall3(string...)) and tuple
>handling (form_heaptuple etc...) and insertion were manually disabled. So
>the only code measured is reading from disk and parsing to the attribute
>level.
>
>
Arguably this might exaggerate the effect quite significantly. Users
will want to know the real time effect on a complete COPY. Depending on
how much the pasing is in the total time your 20% improvement in parsing
might only be a small fraction of 20% improvement in COPY.
Like you, I'm happy we have seen a 5 times improvement in parsing. Is it
possible you can factor out something smallish from your patch that
might make up the balance?
cheers
andrew
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 05:21 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-10 00:41 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-10 01:01 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
@ 2005-08-10 04:39 ` Luke Lonergan <llonergan@greenplum.com>
0 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-08-10 04:39 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Andrew,
> Arguably this might exaggerate the effect quite significantly. Users
> will want to know the real time effect on a complete COPY. Depending on
> how much the pasing is in the total time your 20% improvement in parsing
> might only be a small fraction of 20% improvement in COPY.
Arguably has already been argued. We knew this because we profiled the
entire COPY process and came up with this approx. breakdown for a specific
case:
Parsing: 25%
Attribute Conversion: 40%
Data Insertion: 35%
Net copy rate: 8 MB/s on a filesystem that does 240 MB/s
So - if we speed up parsing by 500% or 450%, the end result is about a
20-30% speed increase in the overall process.
Note that we're still a *long* way from getting anywhere near the limit of
the I/O subsystem at 12 MB/s. Oracle can probably get 5-8 times this data
rate, if not better.
The attribute conversion logic is also very slow and needs similar
improvements.
The reason we focused first on Parsing is that our MPP version of Bizgres
will reach data loading rates approaching the parsing speed, so we needed to
improve that part to get it out of the way.
We will continue to improve COPY speed in Bizgres so that we can provide
comparable COPY performance to Oracle and MySQL.
> Like you, I'm happy we have seen a 5 times improvement in parsing. Is it
> possible you can factor out something smallish from your patch that
> might make up the balance?
That parsing was 25% of the workload was traceable to a 3 main things:
1) Per character acquisition from source instead of buffering
2) Frequent interruption of the parsing pipeline to handle attribute
conversion
3) Lack of micro-parallelism in the character finding logic
Tom's patch took our contribution from (1) and (2) and his improvements, and
he rejected (3). The net result is that we lost performance from the lack
of (3) but gained performance from his improvements of (1) and (2).
I believe that re-introducing (3) may bring us from 100 MB/s to 150 MB/s
parsing speed.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-10 04:48 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-10 08:15 ` Re: COPY FROM performance improvements Simon Riggs <simon@2ndquadrant.com>
1 sibling, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-08-10 04:48 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Tom,
> As best I can tell, my version of CopyReadAttributes is significantly
> quicker than Alon's, approximately balancing out the fact that my
> version of CopyReadLine is slower. I did the latter first, and would
> now be tempted to rewrite it in the same style as CopyReadAttributes,
> ie one pass of memory-to-memory copy using pointers rather than buffer
> indexes.
I think you are right, with the exception that Alon's results prove out that
the net result of your patch is 20% slower than his.
I think with your speedup of CopyReadAttributes and some additional work on
CopyReadLine the net result could be 50% faster than Alon's patch.
The key thing that is missing is the lack of micro-parallelism in the
character processing in this version. By "inverting the loop", or putting
the characters into a buffer on the outside, then doing fast character
scanning inside with special "fix-up" cases, we exposed long runs of
pipeline-able code to the compiler.
I think there is another way to accomplish the same thing and still preserve
the current structure, but it requires "strip mining" the character buffer
into chunks that can be processed with an explicit loop to check for the
different characters. While it may seem artificial (it is), it will provide
the compiler with the ability to pipeline the character finding logic over
long runs. The other necessary element will have to avoid pipeline stalls
from the "if" conditions as much as possible.
Anyway, thanks for reviewing this code and improving it - it's important to
bring speed increases to our collective customer base. With Bizgres, we're
not satisfied with 12 MB/s, we won't stop until we saturate the I/O bus, so
we may get more extreme with the code than seems reasonable for the general
audience.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-10 04:48 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-08-10 08:15 ` Simon Riggs <simon@2ndquadrant.com>
2005-08-10 15:29 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 52+ messages in thread
From: Simon Riggs @ 2005-08-10 08:15 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
On Tue, 2005-08-09 at 21:48 -0700, Luke Lonergan wrote:
> The key thing that is missing is the lack of micro-parallelism in the
> character processing in this version. By "inverting the loop", or putting
> the characters into a buffer on the outside, then doing fast character
> scanning inside with special "fix-up" cases, we exposed long runs of
> pipeline-able code to the compiler.
>
> I think there is another way to accomplish the same thing and still preserve
> the current structure, but it requires "strip mining" the character buffer
> into chunks that can be processed with an explicit loop to check for the
> different characters. While it may seem artificial (it is), it will provide
> the compiler with the ability to pipeline the character finding logic over
> long runs. The other necessary element will have to avoid pipeline stalls
> from the "if" conditions as much as possible.
This is a key point, IMHO.
That part of the code was specifically written to take advantage of
processing pipelines in the hardware, not because the actual theoretical
algorithm for that approach was itself faster.
Nobody's said what compiler/hardware they have been using, so since both
Alon and Tom say their character finding logic is faster, it is likely
to be down to that? Name your platforms gentlemen, please.
My feeling is that we may learn something here that applies more widely
across many parts of the code.
Best Regards, Simon Riggs
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
2005-08-10 04:48 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-08-10 08:15 ` Re: COPY FROM performance improvements Simon Riggs <simon@2ndquadrant.com>
@ 2005-08-10 15:29 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 52+ messages in thread
From: Tom Lane @ 2005-08-10 15:29 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Luke Lonergan <llonergan@greenplum.com>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org
Simon Riggs <simon@2ndquadrant.com> writes:
> Nobody's said what compiler/hardware they have been using, so since both
> Alon and Tom say their character finding logic is faster, it is likely
> to be down to that? Name your platforms gentlemen, please.
I tested on HPPA with gcc 2.95.3 and on a Pentium 4 with gcc 3.4.3.
Got pretty much the same results on both.
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-07 02:54 ` Luke Lonergan <llonergan@greenplum.com>
2 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-08-07 02:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org
Tom,
The previous timings were for a table with 15 columns of mixed type. We
also test with 1 column to make the parsing overhead more apparent. In the
case of 1 text column with 145MB of input data:
Your patch:
Time: 6612.599 ms
Alon's patch:
Time: 6119.244 ms
Alon's patch is 7.5% faster here, where it was only 3% faster on the 15
column case. This is consistent with a large difference in parsing speed
between your approach and Alon's.
I'm pretty sure that the "mistake" you refer to is responsible for the speed
improvement, and was deliberately chosen to minimize memory copies, etc.
Given that we're looking ahead to getting much higher speeds, approaching
current high performance disk speeds, we've been looking more closely at the
parsing speed. It comes down to a tradeoff between elegant code and speed.
We'll prove it in lab tests soon, where we measure the parsing rate
directly, but these experiments show it clearly, though indirectly.
- Luke
On 8/6/05 2:04 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
> "Alon Goldshuv" <agoldshuv@greenplum.com> writes:
>> New patch attached. It includes very minor changes. These are changes that
>> were committed to CVS 3 weeks ago (copy.c 1.247) which I missed in the
>> previous patch.
>
> I've applied this with (rather extensive) revisions. I didn't like what
> you had done with the control structure --- loading the input buffer
> only at the outermost loop level was a bad design choice IMHO. You had
> sprinkled the code with an unreasonable number of special cases in order
> to try to cope with the effects of that mistake, but there were lots
> of problems still left. Some of the bugs I noticed:
>
> * Broke old-protocol COPY, since that has no provision for stopping at
> the EOF marker except by parsing the data carefully to start with. The
> backend would just hang up unless the total data size chanced to be a
> multiple of 64K.
>
> * Subtle change in interpretation of \. EOF marker (the existing code
> will recognize it even when not at start of line).
>
> * Seems to have thrown away detection of newline format discrepancies.
>
> * Fails for zero-column tables.
>
> * Broke display of column values during error context callback (would
> always show the first column contents no matter which one is being
> complained of).
>
> * DetectLineEnd mistakenly assumes CR mode if very last character of first
> bufferload is CR; need to reserve judgment until next char is available.
>
> * DetectLineEnd fails to account for backslashed control characters,
> so it will e.g. accept \ followed by \n as determining the newline
> style.
>
> * Fails to apply encoding conversion if first line exceeds copy buf
> size, because when DetectLineEnd fails the quick-exit path doesn't do
> it.
>
> * There seem to be several bugs associated with the fact that input_buf[]
> always has 64K of data in it even when EOF has been reached on the
> input. One example:
> echo -n 123 >zzz1
> psql> create temp table t1(f1 text);
> psql> copy t1 from '/home/tgl/zzz1';
> psql> select * from t1;
> hmm ... where'd that 64K of whitespace come from?
>
> I rewrote the patch in a way that retained most of the speedups without
> changing the basic control structure (except for replacing multiple
> CopyReadAttribute calls with one CopyReadAttributes call per line).
>
> I had some difficulty in generating test cases that weren't largely
> I/O-bound, but AFAICT the patch as applied is about the same speed
> as what you submitted.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-15 00:22 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-02 14:59 ` Re: COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Re: COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-07 03:25 ` Luke Lonergan <llonergan@greenplum.com>
2 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-08-07 03:25 UTC (permalink / raw)
To: PostgreSQL-patches <pgsql-patches@postgresql.org>
Tom,
My direct e-mails to you are apparently blocked, so I'll send this to the
list.
I've attached the case we use for load performance testing, with the data
generator modified to produce a single row version of the dataset.
I do believe that you/we will need to invert the processing loop to get the
maximum parsing speed. We will be implementing much higher loading speeds
which require it to compete with Oracle, Netezza, Teradata, so we'll have to
work this out for the best interests of our users.
- Luke
Attachments:
IVP.tgz (application/octet-stream, 13.4K) download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-21 21:55 ` Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
3 siblings, 1 reply; 52+ messages in thread
From: Mark Wong @ 2005-07-21 21:55 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
I just ran through a few tests with the v14 patch against 100GB of data
from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to
give a few details, I only loaded data and started a COPY in parallel
for each the data files:
http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
Here's a visual of my disk layout, for those familiar with the database schema:
http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4-010-dbt3.html
I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
Let me know if you have any questions.
Mark
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-21 23:14 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:47 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
0 siblings, 2 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-07-21 23:14 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
Cool!
At what rate does your disk setup write sequential data, e.g.:
time dd if=/dev/zero of=bigfile bs=8k count=500000
(sized for 2x RAM on a system with 2GB)
BTW - the Compaq smartarray controllers are pretty broken on Linux from a
performance standpoint in our experience. We've had disastrously bad
results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
2.6.10, on the order of 20MB/s.
For comparison, the results on our dual opteron with a single LSI SCSI
controller with software RAID0 on a 2.6.10 kernel:
[llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
count=500000
500000+0 records in
500000+0 records out
real 0m24.702s
user 0m0.077s
sys 0m8.794s
Which calculates out to about 161MB/s.
- Luke
On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
> I just ran through a few tests with the v14 patch against 100GB of data
> from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to
> give a few details, I only loaded data and started a COPY in parallel
> for each the data files:
> http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
>
> Here's a visual of my disk layout, for those familiar with the database
> schema:
> http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4
> -010-dbt3.html
>
> I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
> attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
>
> Let me know if you have any questions.
>
> Mark
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 00:08 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
1 sibling, 1 reply; 52+ messages in thread
From: Joshua D. Drake @ 2005-07-22 00:08 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
Luke Lonergan wrote:
> Cool!
>
> At what rate does your disk setup write sequential data, e.g.:
> time dd if=/dev/zero of=bigfile bs=8k count=500000
>
> (sized for 2x RAM on a system with 2GB)
>
> BTW - the Compaq smartarray controllers are pretty broken on Linux from a
> performance standpoint in our experience. We've had disastrously bad
> results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
> 2.6.10, on the order of 20MB/s.
O.k. this strikes me as interesting, now we know that Compaq and Dell
are borked for Linux. Is there a name brand server (read Enterprise)
that actually does provide reasonable performance?
>
> For comparison, the results on our dual opteron with a single LSI SCSI
> controller with software RAID0 on a 2.6.10 kernel:
>
> [llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
> count=500000
> 500000+0 records in
> 500000+0 records out
>
> real 0m24.702s
> user 0m0.077s
> sys 0m8.794s
>
> Which calculates out to about 161MB/s.
>
> - Luke
>
>
> On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
>
>
>>I just ran through a few tests with the v14 patch against 100GB of data
>>from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to
>>give a few details, I only loaded data and started a COPY in parallel
>>for each the data files:
>>http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
>>
>>Here's a visual of my disk layout, for those familiar with the database
>>schema:
>>http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4
>>-010-dbt3.html
>>
>>I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
>>attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
>>
>>Let me know if you have any questions.
>>
>>Mark
>>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
--
Your PostgreSQL solutions provider, Command Prompt, Inc.
24x7 support - 1.800.492.2240, programming, and consulting
Home of PostgreSQL Replicator, plPHP, plPerlNG and pgPHPToolkit
http://www.commandprompt.com / http://www.postgresql.org
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
@ 2005-07-22 02:04 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
0 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-07-22 02:04 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; +Cc: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
Joshua,
On 7/21/05 5:08 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> O.k. this strikes me as interesting, now we know that Compaq and Dell
> are borked for Linux. Is there a name brand server (read Enterprise)
> that actually does provide reasonable performance?
I think late model Dell (post the bad chipset problem, circa 2001-2?) and
IBM and Sun servers are fine because they all use simple SCSI adapters from
LSI or Adaptec.
The HP Smartarray is an aberration, they don't have good driver support for
Linux and as a consequence have some pretty bad problems with both
performance and stability. On Windows they perform quite well.
Also - there are very big issues with some SATA controllers and Linux we've
seen, particularly the Silicon Image, Highpoint other non-Intel controllers.
Not sure about Nvidia, but the only ones I trust now are 3Ware and the
others mentioned in earlier posts.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 02:53 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Joshua D. Drake @ 2005-07-22 02:53 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
> I think late model Dell (post the bad chipset problem, circa 2001-2?) and
> IBM and Sun servers are fine because they all use simple SCSI adapters from
> LSI or Adaptec.
Well I know that isn't true at least not with ANY of the Dells my
customers have purchased in the last 18 months. They are still really,
really slow.
> Also - there are very big issues with some SATA controllers and Linux we've
> seen, particularly the Silicon Image, Highpoint other non-Intel controllers.
> Not sure about Nvidia, but the only ones I trust now are 3Ware and the
> others mentioned in earlier posts.
I have great success with Silicon Image as long as I am running them
with Linux software RAID. The LSI controllers are also really nice.
J
>
> - Luke
>
>
>
--
Your PostgreSQL solutions provider, Command Prompt, Inc.
24x7 support - 1.800.492.2240, programming, and consulting
Home of PostgreSQL Replicator, plPHP, plPerlNG and pgPHPToolkit
http://www.commandprompt.com / http://www.postgresql.org
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
@ 2005-07-22 04:19 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 05:27 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-07-22 15:03 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
0 siblings, 3 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-07-22 04:19 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; +Cc: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
Joshua,
On 7/21/05 7:53 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> Well I know that isn't true at least not with ANY of the Dells my
> customers have purchased in the last 18 months. They are still really,
> really slow.
That's too bad, can you cite some model numbers? SCSI?
> I have great success with Silicon Image as long as I am running them
> with Linux software RAID. The LSI controllers are also really nice.
That's good to hear, I gave up on Silicon Image controllers on Linux about 1
year ago, which kernel are you using with success? Silicon Image
controllers are the most popular, so it's important to see them supported
well, though I'd rather see more SATA headers than 2 off of the built-in
chipsets.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 05:27 ` Andrew Dunstan <andrew@dunslane.net>
2 siblings, 0 replies; 52+ messages in thread
From: Andrew Dunstan @ 2005-07-22 05:27 UTC (permalink / raw)
To: llonergan@greenplum.com; +Cc: jd@commandprompt.com; markw@osdl.org; andrew@dunslane.net; alvherre@surnet.cl; pgman@candle.pha.pa.us; agoldshuv@greenplum.com; pgsql-patches@postgresql.org; maryedie@osdl.org
this discussion belongs on -performance
cheers
andrew
Luke Lonergan said:
> Joshua,
>
> On 7/21/05 7:53 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
>> Well I know that isn't true at least not with ANY of the Dells my
>> customers have purchased in the last 18 months. They are still really,
>> really slow.
>
> That's too bad, can you cite some model numbers? SCSI?
>
>> I have great success with Silicon Image as long as I am running them
>> with Linux software RAID. The LSI controllers are also really nice.
>
> That's good to hear, I gave up on Silicon Image controllers on Linux
> about 1 year ago, which kernel are you using with success? Silicon
> Image
> controllers are the most popular, so it's important to see them
> supported well, though I'd rather see more SATA headers than 2 off of
> the built-in chipsets.
>
> - Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 15:03 ` Joshua D. Drake <jd@commandprompt.com>
2 siblings, 0 replies; 52+ messages in thread
From: Joshua D. Drake @ 2005-07-22 15:03 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
Luke Lonergan wrote:
> Joshua,
>
> On 7/21/05 7:53 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
>
>>Well I know that isn't true at least not with ANY of the Dells my
>>customers have purchased in the last 18 months. They are still really,
>>really slow.
>
>
> That's too bad, can you cite some model numbers? SCSI?
Yeah I will get them and post, but yes they are all SCSI.
>
>
>>I have great success with Silicon Image as long as I am running them
>>with Linux software RAID. The LSI controllers are also really nice.
>
>
> That's good to hear, I gave up on Silicon Image controllers on Linux about 1
> year ago, which kernel are you using with success?
Any of the 2.6 kernels. ALso the laster 2.4 (+22 I believe) support it
pretty well as well.
Silicon Image
> controllers are the most popular, so it's important to see them supported
> well, though I'd rather see more SATA headers than 2 off of the built-in
> chipsets.
>
> - Luke
>
--
Your PostgreSQL solutions provider, Command Prompt, Inc.
24x7 support - 1.800.492.2240, programming, and consulting
Home of PostgreSQL Replicator, plPHP, plPerlNG and pgPHPToolkit
http://www.commandprompt.com / http://www.postgresql.org
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 16:49 ` Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2 siblings, 1 reply; 52+ messages in thread
From: Patrick Welche @ 2005-07-22 16:49 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Joshua D. Drake <jd@commandprompt.com>; Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-patches@postgresql.org; maryedie@osdl.org
On Thu, Jul 21, 2005 at 09:19:04PM -0700, Luke Lonergan wrote:
> Joshua,
>
> On 7/21/05 7:53 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> > Well I know that isn't true at least not with ANY of the Dells my
> > customers have purchased in the last 18 months. They are still really,
> > really slow.
>
> That's too bad, can you cite some model numbers? SCSI?
I would be interested too, given
http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=30531
Cheers,
Patrick
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
@ 2005-07-22 17:11 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Joshua D. Drake @ 2005-07-22 17:11 UTC (permalink / raw)
To: Patrick Welche <prlw1@newn.cam.ac.uk>; +Cc: Luke Lonergan <llonergan@greenplum.com>; Mark Wong <markw@osdl.org>; Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-performance; maryedie@osdl.org
Here is the SCSI output:
Web Server
SCSI subsystem driver Revision: 1.00
megaraid: v1.18j (Release Date: Mon Jul 7 14:39:55 EDT 2003)
megaraid: found 0x1028:0x000f:idx 0:bus 4:slot 3:func 0
scsi0 : Found a MegaRAID controller at 0xf883f000, IRQ: 18
scsi0 : Enabling 64 bit support
megaraid: [412W:H406] detected 1 logical drives
megaraid: supports extended CDBs.
megaraid: channel[1] is raid.
megaraid: channel[2] is raid.
scsi0 : LSI Logic MegaRAID 412W 254 commands 15 targs 5 chans 7 luns
Database Server
SCSI subsystem driver Revision: 1.00
megaraid: v1.18j (Release Date: Mon Jul 7 14:39:55 EDT 2003)
megaraid: found 0x101e:0x1960:idx 0:bus 5:slot 0:func 0
scsi0 : Found a MegaRAID controller at 0xf883f000, IRQ: 21
scsi0 : Enabling 64 bit support
megaraid: [196T:3.33] detected 1 logical drives
megaraid: supports extended CDBs.
megaraid: channel[1] is raid.
megaraid: channel[2] is raid.
scsi0 : LSI Logic MegaRAID 196T 254 commands 15 targs 5 chans 7 luns
Starting timer : 0 0
blk: queue c5f2d218, I/O limit 4095Mb (mask 0xffffffff)
scsi0: scanning virtual channel 0 for logical drives.
Vendor: MegaRAID Model: LD 0 RAID5 86G Rev: 196T
Type: Direct-Access ANSI SCSI revision: 02
Starting timer : 0 0
The webserver is a 1U and it actually performs better on the IO than the
database server even though the database server is running 6 disks versus 3.
The database server is a PE (Power Edge) 6600
Database Server IO:
[root@master root]# /sbin/hdparm -tT /dev/sda
/dev/sda:
Timing buffer-cache reads: 1888 MB in 2.00 seconds = 944.00 MB/sec
Timing buffered disk reads: 32 MB in 3.06 seconds = 10.46 MB/sec
Second Database Server IO:
[root@pq-slave root]# /sbin/hdparm -tT /dev/sda
/dev/sda:
Timing buffer-cache reads: 1816 MB in 2.00 seconds = 908.00 MB/sec
Timing buffered disk reads: 26 MB in 3.11 seconds = 8.36 MB/sec
[root@pq-slave root]#
Which is just horrible.
Sincerely,
Joshua D. Drake
Patrick Welche wrote:
> On Thu, Jul 21, 2005 at 09:19:04PM -0700, Luke Lonergan wrote:
>
>>Joshua,
>>
>>On 7/21/05 7:53 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
>>
>>>Well I know that isn't true at least not with ANY of the Dells my
>>>customers have purchased in the last 18 months. They are still really,
>>>really slow.
>>
>>That's too bad, can you cite some model numbers? SCSI?
>
>
> I would be interested too, given
>
> http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=30531
>
>
> Cheers,
>
> Patrick
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
--
Your PostgreSQL solutions company - Command Prompt, Inc. 1.800.492.2240
PostgreSQL Replication, Consulting, Custom Programming, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
@ 2005-07-22 19:28 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
0 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-07-22 19:28 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; +Cc: Mark Wong <markw@osdl.org>; pgsql-performance; maryedie@osdl.org
Joshua,
On 7/22/05 10:11 AM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> The database server is a PE (Power Edge) 6600
>
> Database Server IO:
>
> [root@master root]# /sbin/hdparm -tT /dev/sda
>
> /dev/sda:
> Timing buffer-cache reads: 1888 MB in 2.00 seconds = 944.00 MB/sec
> Timing buffered disk reads: 32 MB in 3.06 seconds = 10.46 MB/sec
>
> Second Database Server IO:
>
> [root@pq-slave root]# /sbin/hdparm -tT /dev/sda
>
> /dev/sda:
> Timing buffer-cache reads: 1816 MB in 2.00 seconds = 908.00 MB/sec
> Timing buffered disk reads: 26 MB in 3.11 seconds = 8.36 MB/sec
> [root@pq-slave root]#
Can you post the "time dd if=/dev/zero of=bigfile bs=8k count=500000"
results? Also do the reverse (read the file) with "time dd if=bigfile
of=/dev/null bs=8k".
I think you are observing what we've known for a while, hardware RAID is
horribly slow. We've not found a hardware RAID adapter of this class yet
that shows reasonable read or write performance. The Adaptec 2400R or the
LSI or others have terrible internal I/O compared to raw SCSI with software
RAID, and even the CPU usage is higher on these cards while doing slower I/O
than linux SW RAID.
Notably - we've found that the 3Ware RAID controller does a better job than
the low end SCSI RAID at HW RAID support, and also exports JBOD at high
speeds. If you export JBOD on the low end SCSI RAID adapters, the
performance is also very poor, though generally faster than using HW RAID.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-28 23:43 ` Mark Wong <markw@osdl.org>
2005-07-28 23:57 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-29 04:07 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 2 replies; 52+ messages in thread
From: Mark Wong @ 2005-07-28 23:43 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
On Fri, 22 Jul 2005 12:28:43 -0700
"Luke Lonergan" <llonergan@greenplum.com> wrote:
> Joshua,
>
> On 7/22/05 10:11 AM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> > The database server is a PE (Power Edge) 6600
> >
> > Database Server IO:
> >
> > [root@master root]# /sbin/hdparm -tT /dev/sda
> >
> > /dev/sda:
> > Timing buffer-cache reads: 1888 MB in 2.00 seconds = 944.00 MB/sec
> > Timing buffered disk reads: 32 MB in 3.06 seconds = 10.46 MB/sec
> >
> > Second Database Server IO:
> >
> > [root@pq-slave root]# /sbin/hdparm -tT /dev/sda
> >
> > /dev/sda:
> > Timing buffer-cache reads: 1816 MB in 2.00 seconds = 908.00 MB/sec
> > Timing buffered disk reads: 26 MB in 3.11 seconds = 8.36 MB/sec
> > [root@pq-slave root]#
>
> Can you post the "time dd if=/dev/zero of=bigfile bs=8k count=500000"
> results? Also do the reverse (read the file) with "time dd if=bigfile
> of=/dev/null bs=8k".
>
> I think you are observing what we've known for a while, hardware RAID is
> horribly slow. We've not found a hardware RAID adapter of this class yet
> that shows reasonable read or write performance. The Adaptec 2400R or the
> LSI or others have terrible internal I/O compared to raw SCSI with software
> RAID, and even the CPU usage is higher on these cards while doing slower I/O
> than linux SW RAID.
>
> Notably - we've found that the 3Ware RAID controller does a better job than
> the low end SCSI RAID at HW RAID support, and also exports JBOD at high
> speeds. If you export JBOD on the low end SCSI RAID adapters, the
> performance is also very poor, though generally faster than using HW RAID.
Are there any recommendations for Qlogic controllers on Linux, scsi or
fiber channel? I might be able to my hands on some. I have pci-x slots
for AMD, Itanium, or POWER5 if the architecture makes a difference.
Mark
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-28 23:57 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-29 04:03 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
1 sibling, 1 reply; 52+ messages in thread
From: Joshua D. Drake @ 2005-07-28 23:57 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: Luke Lonergan <llonergan@greenplum.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
>>
>>>[root@pq-slave root]# /sbin/hdparm -tT /dev/sda
>>>
>>>/dev/sda:
>>> Timing buffer-cache reads: 1816 MB in 2.00 seconds = 908.00 MB/sec
>>> Timing buffered disk reads: 26 MB in 3.11 seconds = 8.36 MB/sec
>>>[root@pq-slave root]#
>>
>>Can you post the "time dd if=/dev/zero of=bigfile bs=8k count=500000"
>>results? Also do the reverse (read the file) with "time dd if=bigfile
>>of=/dev/null bs=8k".
I didn't see this come across before... here ya go:
time dd if=/dev/zero of=bigfile bs=8k count=500000
500000+0 records in
500000+0 records out
real 1m52.738s
user 0m0.310s
sys 0m36.590s
time dd if=bigfile of=/dev/null bs=8k
time dd if=bigfile of=/dev/null bs=8k
500000+0 records in
500000+0 records out
real 4m38.742s
user 0m0.320s
sys 0m27.870s
FYI on your hardware raid comment... I easily get 50 megs a second on my
3ware controllers and faster on my LSI SATA controllers.
Sincerely,
Joshua D. Drake
>
--
Your PostgreSQL solutions provider, Command Prompt, Inc.
24x7 support - 1.800.492.2240, programming, and consulting
Home of PostgreSQL Replicator, plPHP, plPerlNG and pgPHPToolkit
http://www.commandprompt.com / http://www.postgresql.org
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-28 23:57 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
@ 2005-07-29 04:03 ` Luke Lonergan <llonergan@greenplum.com>
0 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-07-29 04:03 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; Mark Wong <markw@osdl.org>; +Cc: Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
>>> Can you post the "time dd if=/dev/zero of=bigfile bs=8k count=500000"
>>> results? Also do the reverse (read the file) with "time dd if=bigfile
>>> of=/dev/null bs=8k".
>
> I didn't see this come across before... here ya go:
>
> time dd if=/dev/zero of=bigfile bs=8k count=500000
>
> 500000+0 records in
> 500000+0 records out
>
> real 1m52.738s
> user 0m0.310s
> sys 0m36.590s
So, that's 35MB/s, or 1/2 of a single disk drive.
> time dd if=bigfile of=/dev/null bs=8k
>
> time dd if=bigfile of=/dev/null bs=8k
> 500000+0 records in
> 500000+0 records out
>
> real 4m38.742s
> user 0m0.320s
> sys 0m27.870s
And that's 14MB/s, or < 1/4 of a single disk drive.
> FYI on your hardware raid comment... I easily get 50 megs a second on my
> 3ware controllers and faster on my LSI SATA controllers.
Then you are almost getting one disk worth of bandwidth.
By comparison, we get this using Linux software RAID on Xeon or Opteron:
$ time dd if=/dev/zero of=bigfile bs=8k count=500000
500000+0 records in
500000+0 records out
real 0m26.927s
user 0m0.074s
sys 0m8.769s
$ time dd if=bigfile of=/dev/null bs=8k
500000+0 records in
500000+0 records out
real 0m28.190s
user 0m0.039s
sys 0m8.349s
with less CPU usage than HW SCSI RAID controllers.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-29 04:07 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-29 12:37 ` Re: [PATCHES] COPY FROM performance improvements Bruce Momjian <pgman@candle.pha.pa.us>
1 sibling, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-07-29 04:07 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
Mark,
On 7/28/05 4:43 PM, "Mark Wong" <markw@osdl.org> wrote:
> Are there any recommendations for Qlogic controllers on Linux, scsi or
> fiber channel? I might be able to my hands on some. I have pci-x slots
> for AMD, Itanium, or POWER5 if the architecture makes a difference.
I don't have a recommendation for a particular one, it's been too long
(1998) since I've used one with Linux. However, I'd like to see a
comparison between Emulex and Qlogic and a winner chosen. We've had some
apparent driver issues with a client running Emulex on Linux, even using
many different versions of the kernel.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-29 04:07 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-29 12:37 ` Bruce Momjian <pgman@candle.pha.pa.us>
2005-07-29 17:54 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Bruce Momjian @ 2005-07-29 12:37 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Mark Wong <markw@osdl.org>; Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
Luke Lonergan wrote:
> Mark,
>
> On 7/28/05 4:43 PM, "Mark Wong" <markw@osdl.org> wrote:
>
> > Are there any recommendations for Qlogic controllers on Linux, scsi or
> > fiber channel? I might be able to my hands on some. I have pci-x slots
> > for AMD, Itanium, or POWER5 if the architecture makes a difference.
>
> I don't have a recommendation for a particular one, it's been too long
> (1998) since I've used one with Linux. However, I'd like to see a
> comparison between Emulex and Qlogic and a winner chosen. We've had some
> apparent driver issues with a client running Emulex on Linux, even using
> many different versions of the kernel.
Where is the most recent version of the COPY patch?
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-29 04:07 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-29 12:37 ` Re: [PATCHES] COPY FROM performance improvements Bruce Momjian <pgman@candle.pha.pa.us>
@ 2005-07-29 17:54 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-01 16:32 ` Re: [PATCHES] COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 52+ messages in thread
From: Luke Lonergan @ 2005-07-29 17:54 UTC (permalink / raw)
To: Bruce Momjian <pgman@candle.pha.pa.us>; +Cc: Mark Wong <markw@osdl.org>; Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
Bruce,
On 7/29/05 5:37 AM, "Bruce Momjian" <pgman@candle.pha.pa.us> wrote:
> Where is the most recent version of the COPY patch?
My direct e-mails aren't getting to you, they are trapped in a spam filter
on your end, so you didn't get my e-mail with the patch!
I've attached it here, sorry to the list owner for the patch inclusion /
off-topic.
- Luke
Attachments:
copy_parse_improvements_V15.patch.gz (application/octet-stream, 18.4K) download
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Re: COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 16:49 ` Re: COPY FROM performance improvements Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Re: [PATCHES] COPY FROM performance improvements Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-29 04:07 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-29 12:37 ` Re: [PATCHES] COPY FROM performance improvements Bruce Momjian <pgman@candle.pha.pa.us>
2005-07-29 17:54 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-08-01 16:32 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 52+ messages in thread
From: Tom Lane @ 2005-08-01 16:32 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Bruce Momjian <pgman@candle.pha.pa.us>; Mark Wong <markw@osdl.org>; Joshua D. Drake <jd@commandprompt.com>; Patrick Welche <prlw1@newn.cam.ac.uk>; pgsql-performance; maryedie@osdl.org
"Luke Lonergan" <llonergan@greenplum.com> writes:
> On 7/29/05 5:37 AM, "Bruce Momjian" <pgman@candle.pha.pa.us> wrote:
>> Where is the most recent version of the COPY patch?
> I've attached it here, sorry to the list owner for the patch inclusion /
> off-topic.
This patch appears to reverse out the most recent committed changes in
copy.c.
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
@ 2005-07-22 19:47 ` Mark Wong <markw@osdl.org>
2005-07-23 04:09 ` Re: [PATCHES] COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
1 sibling, 1 reply; 52+ messages in thread
From: Mark Wong @ 2005-07-22 19:47 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-performance; maryedie@osdl.org
On a single spindle:
$ time dd if=/dev/zero of=bigfile bs=8k count=2000000
2000000+0 records in
2000000+0 records out
real 2m8.569s
user 0m0.725s
sys 0m19.633s
None of my drives are partitioned big enough for me to create 2x RAM
sized files on a single disk. I have 16MB RAM and only 36GB drives.
But here are some number for my 12-disk lvm2 striped volume.
$ time dd if=/dev/zero of=bigfile3 bs=8k count=4000000
4000000+0 records in
4000000+0 records out
real 1m17.059s
user 0m1.479s
sys 0m41.293s
Mark
On Thu, 21 Jul 2005 16:14:47 -0700
"Luke Lonergan" <llonergan@greenplum.com> wrote:
> Cool!
>
> At what rate does your disk setup write sequential data, e.g.:
> time dd if=/dev/zero of=bigfile bs=8k count=500000
>
> (sized for 2x RAM on a system with 2GB)
>
> BTW - the Compaq smartarray controllers are pretty broken on Linux from a
> performance standpoint in our experience. We've had disastrously bad
> results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
> 2.6.10, on the order of 20MB/s.
>
> For comparison, the results on our dual opteron with a single LSI SCSI
> controller with software RAID0 on a 2.6.10 kernel:
>
> [llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
> count=500000
> 500000+0 records in
> 500000+0 records out
>
> real 0m24.702s
> user 0m0.077s
> sys 0m8.794s
>
> Which calculates out to about 161MB/s.
>
> - Luke
>
>
> On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
>
> > I just ran through a few tests with the v14 patch against 100GB of data
> > from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to
> > give a few details, I only loaded data and started a COPY in parallel
> > for each the data files:
> > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
> >
> > Here's a visual of my disk layout, for those familiar with the database
> > schema:
> > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4
> > -010-dbt3.html
> >
> > I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
> > attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
> >
> > Let me know if you have any questions.
> >
> > Mark
> >
>
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Re: COPY FROM performance improvements Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Re: COPY FROM performance improvements Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Re: COPY FROM performance improvements Luke Lonergan <llonergan@greenplum.com>
2005-07-22 19:47 ` Re: [PATCHES] COPY FROM performance improvements Mark Wong <markw@osdl.org>
@ 2005-07-23 04:09 ` Luke Lonergan <llonergan@greenplum.com>
0 siblings, 0 replies; 52+ messages in thread
From: Luke Lonergan @ 2005-07-23 04:09 UTC (permalink / raw)
To: Mark Wong <markw@osdl.org>; +Cc: Andrew Dunstan <andrew@dunslane.net>; Alvaro Herrera <alvherre@surnet.cl>; Bruce Momjian <pgman@candle.pha.pa.us>; Alon Goldshuv <agoldshuv@greenplum.com>; pgsql-performance; maryedie@osdl.org
Mark,
On 7/22/05 12:47 PM, "Mark Wong" <markw@osdl.org> wrote:
> On a single spindle:
>
> $ time dd if=/dev/zero of=bigfile bs=8k count=2000000
> 2000000+0 records in
> 2000000+0 records out
>
> real 2m8.569s
> user 0m0.725s
> sys 0m19.633s
This is super fast! 124MB/s seems too fast for true write performance on a
single spindle.
> But here are some number for my 12-disk lvm2 striped volume.
So, software striping on how many controllers?
> $ time dd if=/dev/zero of=bigfile3 bs=8k count=4000000
> 4000000+0 records in
> 4000000+0 records out
>
> real 1m17.059s
> user 0m1.479s
> sys 0m41.293s
Again - super fast at 416MB/s. How many controllers?
When we had our problems with the cciss driver and the smartarray 5i/6
controllers, we found the only way to get any performance out of them was to
run them in JBOD mode and software stripe. However, when we did so the CPU
usage skyrocketed and the performance of simple SCSI adapters was 50% faster
with less CPU consumption. These numbers show 100*(42.8/67) = 64% CPU
consumption, I'd expect less with 2 simple U320 SCSI controllers.
- Luke
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PATCHES] COPY FROM performance improvements
@ 2005-08-01 19:48 Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-01 23:51 ` Re: [PERFORM] COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 52+ messages in thread
From: Alon Goldshuv @ 2005-08-01 19:48 UTC (permalink / raw)
To: pgsql-performance; +Cc: Luke Lonergan <llonergan@greenplum.com>
Tom,
>> I've attached it here, sorry to the list owner for the patch inclusion /
>> off-topic.
>
> This patch appears to reverse out the most recent committed changes in
> copy.c.
Which changes do you refer to? I thought I accommodated all the recent
changes (I recall some changes to the tupletable/tupleslot interface, HEADER
in cvs, and hex escapes and maybe one or 2 more). What did I miss?
Thanks.
Alon.
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PERFORM] COPY FROM performance improvements
2005-08-01 19:48 Re: [PATCHES] COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
@ 2005-08-01 23:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-02 15:03 ` Re: [PERFORM] COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
0 siblings, 1 reply; 52+ messages in thread
From: Tom Lane @ 2005-08-01 23:51 UTC (permalink / raw)
To: Alon Goldshuv <agoldshuv@greenplum.com>; +Cc: pgsql-patches@postgresql.org, "Luke Lonergan" <llonergan@greenplum.com>
"Alon Goldshuv" <agoldshuv@greenplum.com> writes:
>> This patch appears to reverse out the most recent committed changes in
>> copy.c.
> Which changes do you refer to? I thought I accommodated all the recent
> changes (I recall some changes to the tupletable/tupleslot interface, HEADER
> in cvs, and hex escapes and maybe one or 2 more). What did I miss?
The latest touch of copy.c, namely this patch:
2005-07-10 17:13 tgl
* doc/src/sgml/ref/create_type.sgml, src/backend/commands/copy.c,
src/backend/commands/typecmds.c, src/backend/tcop/fastpath.c,
src/backend/tcop/postgres.c, src/backend/utils/adt/arrayfuncs.c,
src/backend/utils/adt/date.c, src/backend/utils/adt/numeric.c,
src/backend/utils/adt/rowtypes.c,
src/backend/utils/adt/timestamp.c, src/backend/utils/adt/varbit.c,
src/backend/utils/adt/varchar.c, src/backend/utils/adt/varlena.c,
src/backend/utils/mb/mbutils.c, src/include/catalog/catversion.h,
src/include/catalog/pg_proc.h,
src/test/regress/expected/type_sanity.out,
src/test/regress/sql/type_sanity.sql: Change typreceive function
API so that receive functions get the same optional arguments as
text input functions, ie, typioparam OID and atttypmod. Make all
the datatypes that use typmod enforce it the same way in typreceive
as they do in typinput. This fixes a problem with failure to
enforce length restrictions during COPY FROM BINARY.
It was rather obvious, given that the first chunk of the patch backed up
the file's CVS version stamp from 1.247 to 1.246 :-(
regards, tom lane
^ permalink raw reply [nested|flat] 52+ messages in thread
* Re: [PERFORM] COPY FROM performance improvements
2005-08-01 19:48 Re: [PATCHES] COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-01 23:51 ` Re: [PERFORM] COPY FROM performance improvements Tom Lane <tgl@sss.pgh.pa.us>
@ 2005-08-02 15:03 ` Alon Goldshuv <agoldshuv@greenplum.com>
0 siblings, 0 replies; 52+ messages in thread
From: Alon Goldshuv @ 2005-08-02 15:03 UTC (permalink / raw)
To: pgsql-performance
Tom,
Thanks for pointing it out. I made the small required modifications to match
copy.c version 1.247 and sent it to -patches list. New patch is V16.
Alon.
On 8/1/05 7:51 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
> "Alon Goldshuv" <agoldshuv@greenplum.com> writes:
>>> This patch appears to reverse out the most recent committed changes in
>>> copy.c.
>
>> Which changes do you refer to? I thought I accommodated all the recent
>> changes (I recall some changes to the tupletable/tupleslot interface, HEADER
>> in cvs, and hex escapes and maybe one or 2 more). What did I miss?
>
> The latest touch of copy.c, namely this patch:
>
> 2005-07-10 17:13 tgl
>
> * doc/src/sgml/ref/create_type.sgml, src/backend/commands/copy.c,
> src/backend/commands/typecmds.c, src/backend/tcop/fastpath.c,
> src/backend/tcop/postgres.c, src/backend/utils/adt/arrayfuncs.c,
> src/backend/utils/adt/date.c, src/backend/utils/adt/numeric.c,
> src/backend/utils/adt/rowtypes.c,
> src/backend/utils/adt/timestamp.c, src/backend/utils/adt/varbit.c,
> src/backend/utils/adt/varchar.c, src/backend/utils/adt/varlena.c,
> src/backend/utils/mb/mbutils.c, src/include/catalog/catversion.h,
> src/include/catalog/pg_proc.h,
> src/test/regress/expected/type_sanity.out,
> src/test/regress/sql/type_sanity.sql: Change typreceive function
> API so that receive functions get the same optional arguments as
> text input functions, ie, typioparam OID and atttypmod. Make all
> the datatypes that use typmod enforce it the same way in typreceive
> as they do in typinput. This fixes a problem with failure to
> enforce length restrictions during COPY FROM BINARY.
>
> It was rather obvious, given that the first chunk of the patch backed up
> the file's CVS version stamp from 1.247 to 1.246 :-(
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 52+ messages in thread
end of thread, other threads:[~2005-08-10 15:29 UTC | newest]
Thread overview: 52+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-06-27 06:23 Re: COPY FROM performance improvements Luke Lonergan <LLonergan@greenplum.com>
2005-06-27 11:45 ` Andrew Dunstan <andrew@dunslane.net>
2005-06-27 18:52 ` Luke Lonergan <llonergan@greenplum.com>
2005-06-27 20:20 ` Andrew Dunstan <andrew@dunslane.net>
2005-06-28 04:44 ` Luke Lonergan <llonergan@greenplum.com>
2005-06-28 12:53 ` Andrew Dunstan <andrew@dunslane.net>
2005-06-28 23:14 ` Bruce Momjian <pgman@candle.pha.pa.us>
2005-07-15 00:22 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 15:23 ` Andrew Dunstan <andrew@dunslane.net>
2005-07-19 19:54 ` Mark Wong <markw@osdl.org>
2005-07-19 21:05 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 21:37 ` Mark Wong <markw@osdl.org>
2005-07-19 22:06 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-07-19 22:17 ` Andrew Dunstan <andrew@dunslane.net>
2005-07-19 22:51 ` Mark Wong <markw@osdl.org>
2005-07-20 00:39 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-02 14:59 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-06 21:04 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:33 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-07 04:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 05:21 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-10 00:41 ` Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-10 01:01 ` Andrew Dunstan <andrew@dunslane.net>
2005-08-10 04:39 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-10 04:48 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-10 08:15 ` Simon Riggs <simon@2ndquadrant.com>
2005-08-10 15:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-07 02:54 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-07 03:25 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-21 21:55 ` Mark Wong <markw@osdl.org>
2005-07-21 23:14 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 00:08 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 02:04 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 02:53 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 04:19 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-22 05:27 ` Andrew Dunstan <andrew@dunslane.net>
2005-07-22 15:03 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 16:49 ` Patrick Welche <prlw1@newn.cam.ac.uk>
2005-07-22 17:11 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-22 19:28 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-28 23:43 ` Mark Wong <markw@osdl.org>
2005-07-28 23:57 ` Joshua D. Drake <jd@commandprompt.com>
2005-07-29 04:03 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-29 04:07 ` Luke Lonergan <llonergan@greenplum.com>
2005-07-29 12:37 ` Bruce Momjian <pgman@candle.pha.pa.us>
2005-07-29 17:54 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-01 16:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-07-22 19:47 ` Mark Wong <markw@osdl.org>
2005-07-23 04:09 ` Luke Lonergan <llonergan@greenplum.com>
2005-08-01 19:48 Re: [PATCHES] COPY FROM performance improvements Alon Goldshuv <agoldshuv@greenplum.com>
2005-08-01 23:51 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-08-02 15:03 ` Alon Goldshuv <agoldshuv@greenplum.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox