public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 1/1] In psql \copy from, send data to server in larger chunks.
5+ messages / 5 participants
[nested] [flat]
* [PATCH 1/1] In psql \copy from, send data to server in larger chunks.
@ 2021-02-06 22:10 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Heikki Linnakangas @ 2021-02-06 22:10 UTC (permalink / raw)
Previously, we would send each line as a separate CopyData message.
That's pretty wasteful if the table is narrow, as each CopyData message
has 5 bytes of overhead. For efficiency, buffer up and pack 8 kB of input
data into each CopyData message.
---
src/bin/psql/copy.c | 114 +++++++++++++++++++++++++++-----------------
1 file changed, 69 insertions(+), 45 deletions(-)
diff --git a/src/bin/psql/copy.c b/src/bin/psql/copy.c
index 78f0dc5a507..8c56f2470e2 100644
--- a/src/bin/psql/copy.c
+++ b/src/bin/psql/copy.c
@@ -581,77 +581,101 @@ handleCopyIn(PGconn *conn, FILE *copystream, bool isbinary, PGresult **res)
else
{
bool copydone = false;
+ int buflen;
+ bool at_line_begin = true;
+ if (showprompt)
+ {
+ const char *prompt = get_prompt(PROMPT_COPY, NULL);
+
+ fputs(prompt, stdout);
+ fflush(stdout);
+ }
+
+ /*
+ * In text mode, We have to read the input one line at a time, so that
+ * we can stop reading at the EOF marker (\.). We mustn't read beyond
+ * the EOF marker, because if the data is inlined in a SQL script, we
+ * would eat up the commands after the EOF marker.
+ */
+ buflen = 0;
while (!copydone)
- { /* for each input line ... */
- bool firstload;
- bool linedone;
+ {
+ int linelen;
+ char *fgresult;
- if (showprompt)
+ if (at_line_begin && showprompt)
{
- const char *prompt = get_prompt(PROMPT_COPY, NULL);
+ const char *prompt;
+ sigint_interrupt_enabled = false;
+
+ prompt = get_prompt(PROMPT_COPY, NULL);
fputs(prompt, stdout);
fflush(stdout);
- }
- firstload = true;
- linedone = false;
-
- while (!linedone)
- { /* for each bufferload in line ... */
- int linelen;
- char *fgresult;
-
- /* enable longjmp while waiting for input */
sigint_interrupt_enabled = true;
+ }
- fgresult = fgets(buf, sizeof(buf), copystream);
-
- sigint_interrupt_enabled = false;
+ /* enable longjmp while waiting for input */
+ sigint_interrupt_enabled = true;
- if (!fgresult)
- {
- copydone = true;
- break;
- }
+ fgresult = fgets(&buf[buflen], COPYBUFSIZ - buflen, copystream);
- linelen = strlen(buf);
+ sigint_interrupt_enabled = false;
- /* current line is done? */
- if (linelen > 0 && buf[linelen - 1] == '\n')
- linedone = true;
+ if (!fgresult)
+ copydone = true;
+ else
+ {
+ linelen = strlen(fgresult);
+ buflen += linelen;
- /* check for EOF marker, but not on a partial line */
- if (firstload)
+ if (buf[buflen - 1] == '\n')
{
- /*
- * This code erroneously assumes '\.' on a line alone
- * inside a quoted CSV string terminates the \copy.
- * https://www.postgresql.org/message-id/[email protected]
- */
- if (strcmp(buf, "\\.\n") == 0 ||
- strcmp(buf, "\\.\r\n") == 0)
+ /* check for EOF marker, but not on a partial line */
+ if (at_line_begin)
{
- copydone = true;
- break;
+ /*
+ * This code erroneously assumes '\.' on a line alone
+ * inside a quoted CSV string terminates the \copy.
+ * https://www.postgresql.org/message-id/[email protected]
+ */
+ if ((linelen == 3 && memcmp(fgresult, "\\.\n", 3) == 0) ||
+ (linelen == 4 && memcmp(fgresult, "\\.\r\n", 4) == 0))
+ {
+ copydone = true;
+ }
}
- firstload = false;
+ if (copystream == pset.cur_cmd_source)
+ {
+ pset.lineno++;
+ pset.stmt_lineno++;
+ }
+ at_line_begin = true;
}
+ else
+ at_line_begin = false;
+ }
- if (PQputCopyData(conn, buf, linelen) <= 0)
+ /*
+ * If the buffer is full, or we've reached the EOF, flush it.
+ *
+ * Make sure there's always space for four more bytes in the buffer,
+ * plus a NUL terminator. That way, an EOF marker is never split
+ * across two fgets() calls, which simplies the logic.
+ */
+ if (buflen >= COPYBUFSIZ - 5 || (copydone && buflen > 0))
+ {
+ if (PQputCopyData(conn, buf, buflen) <= 0)
{
OK = false;
copydone = true;
break;
}
- }
- if (copystream == pset.cur_cmd_source)
- {
- pset.lineno++;
- pset.stmt_lineno++;
+ buflen = 0;
}
}
}
--
2.30.0
--------------20DC7BBD34247104D0B00618--
^ permalink raw reply [nested|flat] 5+ messages in thread
* NumericShort vs NumericLong format
@ 2023-03-07 03:46 Amin <[email protected]>
2023-03-07 04:46 ` Re: NumericShort vs NumericLong format David G. Johnston <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Amin @ 2023-03-07 03:46 UTC (permalink / raw)
To: pgsql-hackers
Hi,
- How can I determine which format will be used for a numeric type?
- What the precision and scale values should be for pgsql to use the long
format? Is there a threshold?
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format
2023-03-07 03:46 NumericShort vs NumericLong format Amin <[email protected]>
@ 2023-03-07 04:46 ` David G. Johnston <[email protected]>
2023-03-07 05:15 ` Re: NumericShort vs NumericLong format Tom Lane <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: David G. Johnston @ 2023-03-07 04:46 UTC (permalink / raw)
To: Amin <[email protected]>; +Cc: pgsql-hackers
I'll give this a go as a learning exercise for myself...
On Mon, Mar 6, 2023 at 8:47 PM Amin <[email protected]> wrote:
>
> - How can I determine which format will be used for a numeric type?
>
https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils...
(the three constants are decimal 63, 63, and -64 respectively)
> - What the precision and scale values should be for pgsql to use the long
> format? Is there a threshold?
>
>
Ones that cause the linked-to test to return false I suppose.
David J.
As an aside, for anyone more fluent than I who reads this, is the use of
the word "dynamic scale" in this code comment supposed to be "display
scale"?
https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils...
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format
2023-03-07 03:46 NumericShort vs NumericLong format Amin <[email protected]>
2023-03-07 04:46 ` Re: NumericShort vs NumericLong format David G. Johnston <[email protected]>
@ 2023-03-07 05:15 ` Tom Lane <[email protected]>
2024-04-09 11:41 ` Re: NumericShort vs NumericLong format Ole Peder Brandtzæg <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Tom Lane @ 2023-03-07 05:15 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Amin <[email protected]>; pgsql-hackers
"David G. Johnston" <[email protected]> writes:
> As an aside, for anyone more fluent than I who reads this, is the use of
> the word "dynamic scale" in this code comment supposed to be "display
> scale"?
> https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils...
Yeah, I think you're right.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: NumericShort vs NumericLong format
2023-03-07 03:46 NumericShort vs NumericLong format Amin <[email protected]>
2023-03-07 04:46 ` Re: NumericShort vs NumericLong format David G. Johnston <[email protected]>
2023-03-07 05:15 ` Re: NumericShort vs NumericLong format Tom Lane <[email protected]>
@ 2024-04-09 11:41 ` Ole Peder Brandtzæg <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Ole Peder Brandtzæg @ 2024-04-09 11:41 UTC (permalink / raw)
To: [email protected]
On Tue, Mar 07, 2023 at 12:15:27AM -0500, Tom Lane wrote:
> "David G. Johnston" <[email protected]> writes:
> > As an aside, for anyone more fluent than I who reads this, is the use of
> > the word "dynamic scale" in this code comment supposed to be "display
> > scale"?
> > https://github.com/postgres/postgres/blob/cf96907aadca454c4094819c2ecddee07eafe203/src/backend/utils...
>
> Yeah, I think you're right.
Familiarizing myself with numeric.c today, I too was confused by this.
AFAICT, it's meant to say display scale as used elsewhere in the file;
for instance, the comment for NumericShort's n_header reads "Sign +
display scale + weight". Would it be appropriate if I submitted a patch
for this? It's admittedly trivial, but I figured I should say hi before
submitting one.
All the best,
Ole
--
Ole Peder Brandtzæg | En KLST/ITK-hybrid
Please don't look at me with those eyes
Please don't hint that you're capable of lies
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-04-09 11:41 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-02-06 22:10 [PATCH 1/1] In psql \copy from, send data to server in larger chunks. Heikki Linnakangas <[email protected]>
2023-03-07 03:46 NumericShort vs NumericLong format Amin <[email protected]>
2023-03-07 04:46 ` Re: NumericShort vs NumericLong format David G. Johnston <[email protected]>
2023-03-07 05:15 ` Re: NumericShort vs NumericLong format Tom Lane <[email protected]>
2024-04-09 11:41 ` Re: NumericShort vs NumericLong format Ole Peder Brandtzæg <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox