public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/1] In psql \copy from, send data to server in larger chunks. 5+ messages / 5 participants [nested] [flat]
* [PATCH 1/1] In psql \copy from, send data to server in larger chunks. @ 2021-02-06 22:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Heikki Linnakangas @ 2021-02-06 22:10 UTC (permalink / raw) Previously, we would send each line as a separate CopyData message. That's pretty wasteful if the table is narrow, as each CopyData message has 5 bytes of overhead. For efficiency, buffer up and pack 8 kB of input data into each CopyData message. --- src/bin/psql/copy.c | 114 +++++++++++++++++++++++++++----------------- 1 file changed, 69 insertions(+), 45 deletions(-) diff --git a/src/bin/psql/copy.c b/src/bin/psql/copy.c index 78f0dc5a507..8c56f2470e2 100644 --- a/src/bin/psql/copy.c +++ b/src/bin/psql/copy.c @@ -581,77 +581,101 @@ handleCopyIn(PGconn *conn, FILE *copystream, bool isbinary, PGresult **res) else { bool copydone = false; + int buflen; + bool at_line_begin = true; + if (showprompt) + { + const char *prompt = get_prompt(PROMPT_COPY, NULL); + + fputs(prompt, stdout); + fflush(stdout); + } + + /* + * In text mode, We have to read the input one line at a time, so that + * we can stop reading at the EOF marker (\.). We mustn't read beyond + * the EOF marker, because if the data is inlined in a SQL script, we + * would eat up the commands after the EOF marker. + */ + buflen = 0; while (!copydone) - { /* for each input line ... */ - bool firstload; - bool linedone; + { + int linelen; + char *fgresult; - if (showprompt) + if (at_line_begin && showprompt) { - const char *prompt = get_prompt(PROMPT_COPY, NULL); + const char *prompt; + sigint_interrupt_enabled = false; + + prompt = get_prompt(PROMPT_COPY, NULL); fputs(prompt, stdout); fflush(stdout); - } - firstload = true; - linedone = false; - - while (!linedone) - { /* for each bufferload in line ... */ - int linelen; - char *fgresult; - - /* enable longjmp while waiting for input */ sigint_interrupt_enabled = true; + } - fgresult = fgets(buf, sizeof(buf), copystream); - - sigint_interrupt_enabled = false; + /* enable longjmp while waiting for input */ + sigint_interrupt_enabled = true; - if (!fgresult) - { - copydone = true; - break; - } + fgresult = fgets(&buf[buflen], COPYBUFSIZ - buflen, copystream); - linelen = strlen(buf); + sigint_interrupt_enabled = false; - /* current line is done? */ - if (linelen > 0 && buf[linelen - 1] == '\n') - linedone = true; + if (!fgresult) + copydone = true; + else + { + linelen = strlen(fgresult); + buflen += linelen; - /* check for EOF marker, but not on a partial line */ - if (firstload) + if (buf[buflen - 1] == '\n') { - /* - * This code erroneously assumes '\.' on a line alone - * inside a quoted CSV string terminates the \copy. - * https://www.postgresql.org/message-id/[email protected] - */ - if (strcmp(buf, "\\.\n") == 0 || - strcmp(buf, "\\.\r\n") == 0) + /* check for EOF marker, but not on a partial line */ + if (at_line_begin) { - copydone = true; - break; + /* + * This code erroneously assumes '\.' on a line alone + * inside a quoted CSV string terminates the \copy. + * https://www.postgresql.org/message-id/[email protected] + */ + if ((linelen == 3 && memcmp(fgresult, "\\.\n", 3) == 0) || + (linelen == 4 && memcmp(fgresult, "\\.\r\n", 4) == 0)) + { + copydone = true; + } } - firstload = false; + if (copystream == pset.cur_cmd_source) + { + pset.lineno++; + pset.stmt_lineno++; + } + at_line_begin = true; } + else + at_line_begin = false; + } - if (PQputCopyData(conn, buf, linelen) <= 0) + /* + * If the buffer is full, or we've reached the EOF, flush it. + * + * Make sure there's always space for four more bytes in the buffer, + * plus a NUL terminator. That way, an EOF marker is never split + * across two fgets() calls, which simplies the logic. + */ + if (buflen >= COPYBUFSIZ - 5 || (copydone && buflen > 0)) + { + if (PQputCopyData(conn, buf, buflen) <= 0) { OK = false; copydone = true; break; } - } - if (copystream == pset.cur_cmd_source) - { - pset.lineno++; - pset.stmt_lineno++; + buflen = 0; } } } -- 2.30.0 --------------20DC7BBD34247104D0B00618-- ^ permalink raw reply [nested|flat] 5+ messages in thread
* NumericShort vs NumericLong format @ 2023-03-07 03:46 Amin <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Amin @ 2023-03-07 03:46 UTC (permalink / raw) To: pgsql-hackers Hi, - How can I determine which format will be used for a numeric type? - What the precision and scale values should be for pgsql to use the long format? Is there a threshold? ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format @ 2023-03-07 04:46 David G. Johnston <[email protected]> parent: Amin <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: David G. Johnston @ 2023-03-07 04:46 UTC (permalink / raw) To: Amin <[email protected]>; +Cc: pgsql-hackers I'll give this a go as a learning exercise for myself... On Mon, Mar 6, 2023 at 8:47 PM Amin <[email protected]> wrote: > > - How can I determine which format will be used for a numeric type? > https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils... (the three constants are decimal 63, 63, and -64 respectively) > - What the precision and scale values should be for pgsql to use the long > format? Is there a threshold? > > Ones that cause the linked-to test to return false I suppose. David J. As an aside, for anyone more fluent than I who reads this, is the use of the word "dynamic scale" in this code comment supposed to be "display scale"? https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils... ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format @ 2023-03-07 05:15 Tom Lane <[email protected]> parent: David G. Johnston <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Tom Lane @ 2023-03-07 05:15 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Amin <[email protected]>; pgsql-hackers "David G. Johnston" <[email protected]> writes: > As an aside, for anyone more fluent than I who reads this, is the use of > the word "dynamic scale" in this code comment supposed to be "display > scale"? > https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils... Yeah, I think you're right. regards, tom lane ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format @ 2024-04-09 11:41 Ole Peder Brandtzæg <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Ole Peder Brandtzæg @ 2024-04-09 11:41 UTC (permalink / raw) To: [email protected] On Tue, Mar 07, 2023 at 12:15:27AM -0500, Tom Lane wrote: > "David G. Johnston" <[email protected]> writes: > > As an aside, for anyone more fluent than I who reads this, is the use of > > the word "dynamic scale" in this code comment supposed to be "display > > scale"? > > https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils... > > Yeah, I think you're right. Familiarizing myself with numeric.c today, I too was confused by this. AFAICT, it's meant to say display scale as used elsewhere in the file; for instance, the comment for NumericShort's n_header reads "Sign + display scale + weight". Would it be appropriate if I submitted a patch for this? It's admittedly trivial, but I figured I should say hi before submitting one. All the best, Ole -- Ole Peder Brandtzæg | En KLST/ITK-hybrid Please don't look at me with those eyes Please don't hint that you're capable of lies ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-04-09 11:41 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-02-06 22:10 [PATCH 1/1] In psql \copy from, send data to server in larger chunks. Heikki Linnakangas <[email protected]> 2023-03-07 03:46 NumericShort vs NumericLong format Amin <[email protected]> 2023-03-07 04:46 ` Re: NumericShort vs NumericLong format David G. Johnston <[email protected]> 2023-03-07 05:15 ` Re: NumericShort vs NumericLong format Tom Lane <[email protected]> 2024-04-09 11:41 ` Re: NumericShort vs NumericLong format Ole Peder Brandtzæg <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox